System and method for implementing a collaborative 3d map data fusion platform and virtual world system thereof

Abstract

A method of generating a collaborative 3D map includes providing a 3D map data fusion platform, providing a base satellite map of a world location, publishing a detailed real-time virtual replica network (RVRN) of a real-world location in reference to the base satellite map, obtaining a pose (e.g., position and orientation) of a client device, and performing a simultaneous localization and mapping (SLAM) process in a first coordinate space. The SLAM process uses image data from an imaging device of the client device, the pose of the client device, and / or data of the base satellite map to determine a pose of a plurality of features within the first coordinate space. The method further includes creating a new map including three-dimensional coordinates of the plurality of features within the first coordinate space and merging the new map with the published RVRN, creating a fused 3D map.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of U.S. Provisional Application No. 62 / 955216, filed December 30, 2019, which is incorporated herein by reference. Background Technology

[0003] The use of imaging devices and simultaneous localization and mapping (SLAM) techniques to allow robots or other devices to build maps of unknown environments while tracking their current position within those environments is known in the art. Some SLAM techniques can utilize merged data from multiple imaging devices to generate collaborative SLAM maps.

[0004] Typically, multiple imaging devices are configured on a dedicated mobile platform, such as a drone or robot, responsible for SLAM mapping. Therefore, current collaborative implementations are often geared towards specific devices whose primary task is to build maps of unknown areas, limiting the area where mapping can be done to a smaller number of devices, as the technology is not usually enabled for general user devices. Furthermore, the accuracy of SLAM processes utilizing multiple imaging devices is considered a limiting factor for current collaborative SLAM technologies, as the position and orientation of some devices may not be accurately known during map building, thus reducing the accuracy and reliability of the created map. Additionally, current SLAM maps often limit their application to navigation, and in some cases, to further overlaying objects on the generated map to create augmented reality experiences, thus often ignoring other application areas. Summary of the Invention

[0005] This summary is provided to introduce, in a simplified form, some concepts that will be further described in the detailed embodiments below. This summary is not intended to identify key features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter.

[0006] This disclosure generally relates to computer systems and methods, and more specifically, to a system and method for implementing an updatable (e.g., configured for continuous, intermittent, periodic, etc.) collaborative 3D map data fusion platform and its persistent virtual world system.

[0007] One or more drawbacks described in the background art are addressed by systems and methods for implementing an updatable collaborative 3D map data fusion platform and its persistent virtual world system. In one aspect, the method of this disclosure includes the steps of: providing a 3D map data fusion platform in the memory of one or more server computers including at least one processor; providing a base satellite map of a world location in the 3D map data fusion platform; publishing a detailed real-time virtual copy network (RVRN) of real-world locations in the 3D map data fusion platform with reference to the base satellite map (e.g., aligning the coordinates of the base satellite map, if available); acquiring the pose (e.g., position and orientation) of a first client device; and performing a synchronized localization and mapping (SLAM) process in a first coordinate space. The SLAM process uses image data from at least one imaging device of the first client device, the pose of the first client device, and / or data from the base satellite map to determine a pose having a first plurality of features within the first coordinate space.

[0008] In one embodiment, the method further includes: creating a first new map, the first new map comprising three-dimensional coordinates having a first plurality of features in a first coordinate space; and merging the first new map with the published RVRN to create a fused 3D map. In one embodiment, creating the first new map includes comparing the first plurality of features of the map to the published RVRN using a feature matching algorithm via the 3D map data fusion platform. In one embodiment, the comparison step is performed once a feature threshold number in the first coordinate space is reached (e.g., once a predetermined percentage of a specific area has been mapped and the corresponding features have been obtained). This helps avoid a situation where, if too few features are obtained, the sample size is too small for comparison.

[0009] Therefore, because the newly created map is integrated with the precise 3D geometry that is part of the published detailed RVRN, or because the newly created map provides further 3D geometry and features to the published detailed RVRN, the integrated 3D map includes a higher density of detail and features than the base map. This can support its use in several applications, such as augmented reality or virtual reality. Furthermore, using the base map as a reference for creating a richly detailed RVRN, and using the RVRN as a reference for feature matching in the newly created map, can save computational power and time during the creation of new maps, as areas and / or features already included in the RVRN or base map may not need to be redrawn.

[0010] The system can perform one or more checks to determine whether the RVRN has included features of the newly drawn map. In one embodiment, if the published detailed RVRN includes features of the newly drawn map, the SLAM process can terminate. However, if the published RVRN does not include features of the newly drawn map, the SLAM process can continue until a first new map in the first coordinate space is created.

[0011] In some embodiments, the method further includes obtaining the pose (e.g., position and orientation) of the second client device and then initiating a SLAM process in a second coordinate space. In one embodiment, the SLAM process in the second coordinate space uses image data from at least one imaging device of the second client device, the pose of the second client device, and / or data from the fused 3D map to determine a pose having a second plurality of features within the second coordinate space. In one embodiment, the method further includes creating a second new map including the second plurality of features and merging the second new map with the fused 3D map to create a cooperative 3D map.

[0012] In one embodiment, once sufficient features in the second coordinate space have been mapped, the method continues by using a 3D map data fusion platform and a feature matching algorithm to compare a second set of features from the drawn map with the fused 3D map. If the fused 3D map includes features from the newly drawn map, the SLAM process can terminate. However, if the fused 3D map does not include features from the newly drawn map, the SLAM process can continue until a second map is created.

[0013] In some other embodiments, the method further includes sharing a collaborative 3D map with a persistent virtual world system that includes virtual copies, purely virtual objects, applications, or combinations thereof, wherein objects of the newly drawn map are added to the persistent virtual world system (e.g., as virtual copies).

[0014] In some embodiments, image data is received (e.g., synchronously) from first and second client devices. In such embodiments, the method may include obtaining the poses of the first and second client devices, which may occur after providing a 3D map data fusion platform, a base satellite map, and publishing one or more RVRNs. The method may include performing a SLAM process in first and second coordinate spaces simultaneously, including capturing image data from at least one imaging device of the first and second client devices, wherein the SLAM process is performed by determining poses with first and second plurality of features in the first and second coordinate spaces using the poses of the first and second client devices. The method may include checking whether enough features have been extracted such that feature maps can be compared and matched with each other. The method may include determining whether the first and second plurality of features match (e.g., precisely or to a predetermined extent), and if they match, merging the first and second feature maps, wherein the merging is performed in alignment with the base map. In one embodiment, the method continues to generate a cooperative 3D map by comparing and merging the combined map with the RVRN. If the first and second plurality of features do not match, the method may continue to generate a cooperative 3D map by comparing and merging the first and second feature maps with the RVRN, respectively.

[0015] In some embodiments, the published RVRN is created using computer-aided drafting (CAD) modeling techniques or scanned using one or more computer vision scanning methods. For example, the memory of at least one server computer may include a virtual copy editor, which may include software, hardware, and / or firmware configured to enable users to model and edit virtual copies of real-world entities. This virtual copy editor may be CAD software capable of storing data and instructions required for inputting and editing the virtual copies. The virtual copy editor may allow input of explicit data and instructions associated with each digital copy, such as data and instructions describing the shape, location, position and orientation, physical properties, and expected functions and effects of each copy and the entire system.

[0016] In some embodiments, the SLAM process is partially performed by one or more processors of a client device. In some embodiments, the client device is one or more of a mobile device, smart contact lenses, a head-mounted display, a laptop, a desktop computer, a camera, a drone, and a vehicle. The 3D map data fusion platform fuses multiple maps, including those with multiple features, such as a newly created first map and map data included in an RVRN, through feature comparison and merging algorithms to form a fused 3D map, or a fused 3D map and at least a second map. Furthermore, the 3D map data fusion platform enables multiple client devices to provide their map data for application of such algorithms to merge multiple maps, thereby increasing the number of data points available for the platform to generate collaborative 3D maps.

[0017] In a further embodiment, the method includes obtaining an identifier from a first or second client device; associating the identifier with a first or second new map; and storing the map associated with the identifier in a distributed ledger. The stored identified map can be used, for example, in a reward system that includes providing rewards related to map area contributions by the first or second client device. The reward system can be managed by rules of a smart contract stored in the distributed ledger. The identifier can be, for example, a QR code, URL, IP address, MAC address, cryptographic hash, universally unique identifier, or organization-unique identifier. The reward can be, for example, cryptocurrency, coupons, access to other levels of one or more applications, etc.

[0018] In some embodiments, a base map is obtained using satellite mapping techniques that utilize satellite imagery. In some embodiments, the base map refers to a general version of a map, such as a map of an area that does not include a dense number of features and details, and some details of these features may be omitted. In some embodiments, the base map is obtained using satellite mapping techniques that utilize satellite imagery. In some embodiments, the base map is a 2D map of an area. For example, the base map might be generated by OpenStreetMap. TM Google Maps TM or Crowdmap TM The provided map.

[0019] In some embodiments, the position and orientation of the client device are obtained through a Global Navigation Satellite System (GNSS), mmW geolocation method, internal positioning method, or a combination thereof.

[0020] In some embodiments, the method further includes selecting one or more virtual objects via a first or second client device; and having the first or second client device overlay the selected virtual objects onto one or more locations using a fused 3D map or a collaborative virtual world system as a reference for alignment of the virtual objects.

[0021] In some embodiments, the method further includes adding per-user rights information to the image data, wherein only authorized users have the right to view the image data at that location.

[0022] In some embodiments, the method further includes adding logic, virtual data, and models to at least some virtual objects in the persistent virtual world system to provide self-computation capabilities and autonomous behavior, wherein the models include one or more of 3D models, dynamic models, geometric models, or machine learning models, or combinations thereof. In still other embodiments, the persistent virtual world system is used for autonomous or semi-autonomous management of manufacturing operations, traffic behavior, utilities, or combinations thereof.

[0023] In one embodiment, a system for implementing an updatable 3D spatial map includes at least one server computer comprising a memory and at least one processor, the memory storing a 3D map data fusion platform, wherein the at least one server computer is configured to: provide a basic satellite map of a world location in the 3D map data fusion platform; publish RVRNs of real-world locations on the 3D map data fusion platform with reference to the basic satellite map; acquire the pose of a first client device; initiate a SLAM process in a first coordinate space, wherein the SLAM process uses image data from at least one imaging device of the first client device and the pose of the first client device to determine a pose having a first plurality of features in the first coordinate space; create a first new map including 3D coordinates having the first plurality of features in the first coordinate space; and have the 3D map data fusion platform merge the first new map including 3D coordinates having the first plurality of features in the first coordinate space with the published RVRNs to create a fused 3D map.

[0024] In one embodiment, creating a first new map includes comparing a plurality of features with a published RVRN using one or more feature matching algorithms via a 3D map data fusion platform. The comparison step can proceed once a threshold number of features in the first coordinate space is reached.

[0025] In some embodiments, the client device includes a memory, at least one processor, at least one imaging device configured to capture image data of a location, and a tracking system that provides the position and orientation of the client device.

[0026] According to one embodiment, the 3D map data fusion platform includes instructions that, when executed by at least one processor, trigger at least one processor to initiate a SLAM process.

[0027] In one embodiment, the at least one server computer is further configured to: initiate a SLAM process in a second coordinate space, the process using image data from at least one imaging device of a second client device and the pose of the second client device to determine a pose having a second plurality of features in the second coordinate space; create a second new map including the second plurality of features; and merge the second new map with a fused 3D map to create a collaborative 3D map.

[0028] In one embodiment, the at least one server computer is further configured to, once sufficient features of the second coordinate space have been mapped, compare a second plurality of features of the drawn map with the fused 3D map using a 3D map data fusion platform and a feature matching algorithm; if the fused 3D map includes features of the newly drawn map, the SLAM process ends; if the fused 3D map does not include features of the newly drawn map, the SLAM process continues until a second map is created.

[0029] In some embodiments, the system further includes a second client device located at another location, wherein the 3D map data fusion platform further includes instructions that, when executed by at least one processor, trigger at least one processor to initiate a SLAM process in a second coordinate space by capturing image data from at least one imaging device of the second client device, wherein the SLAM process is performed by determining a pose with a second plurality of features in the second coordinate space using the pose of the second client device; once sufficient features in the second coordinate space have been used to draw a map, the second plurality of features of the drawn map are compared with the fused 3D map using a feature matching algorithm by the 3D map data fusion platform; if the fused 3D map includes features of the newly drawn map, the SLAM process ends; if the fused 3D map does not include features of the newly drawn map, the SLAM process continues until a second map is created; and the collaborative 3D map data fusion platform merges the created second map in the second coordinate space with the fused 3D map to create a collaborative 3D map.

[0030] According to one embodiment, a collaborative 3D map is shared with a persistent virtual world system that includes virtual copies, purely virtual objects, applications, or combinations thereof, wherein objects of the newly drawn map are added to the persistent virtual world system (e.g., as virtual copies).

[0031] According to one embodiment, the at least one server computer is further configured to: obtain an identification code from a first or second client device; associate the identification code with a first new map; and store the identified first new map in a smart contract implemented in a distributed ledger. In one embodiment, the stored identified map is used for a reward system that includes providing rewards related to map area contributions from the first or second client device.

[0032] According to one embodiment, the client device is configured to select one or more virtual objects stored in the memory of at least one server computer; and to overlay the selected virtual objects on one or more locations using a collaborative 3D map shared with a persistent virtual world system as a reference for virtual object alignment.

[0033] The foregoing summary does not include an exhaustive list of all aspects of this disclosure. It is contemplated that this disclosure encompasses all systems and methods that can be practiced from all suitable combinations of the aspects outlined above, as well as all systems and methods disclosed in the following detailed description, particularly those specifically pointed out in the claims filed with this application. Such combinations have particular advantages not specifically described in the foregoing summary. Other features and advantages of this disclosure will become apparent from the accompanying drawings and the following detailed description. Attached Figure Description

[0034] Since the foregoing aspects and many accompanying advantages of the invention will be better understood when taken in conjunction with the accompanying drawings and by referring to the following specific embodiments, the foregoing aspects and many accompanying advantages will become more readily apparent, wherein:

[0035] The specific features, aspects, and advantages of this disclosure will be better understood with reference to the following description and accompanying drawings, in which:

[0036] Figure 1 A schematic diagram of a system for implementing an updatable collaborative 3D map data fusion platform and its persistent virtual world system, according to one embodiment, is depicted.

[0037] Figure 2 A schematic diagram depicts another embodiment of a system for realizing an updatable collaborative 3D map data fusion platform and its persistent virtual world system;

[0038] Figure 3 A schematic diagram depicts another embodiment of a system for realizing an updatable collaborative 3D map data fusion platform and its persistent virtual world system;

[0039] Figure 4 A schematic diagram depicts a SLAM scan performed by two or more client devices and the data flow to the server according to one embodiment;

[0040] Figure 5 A schematic diagram of a persistent virtual world system according to one embodiment is depicted;

[0041] Figure 6 Another schematic diagram depicts a persistent virtual world system according to one embodiment and its relationship with a 3D map data fusion platform;

[0042] Figure 7 A block diagram depicts a method for implementing an updatable collaborative 3D map data fusion platform and its persistent virtual world system according to one embodiment;

[0043] Figure 8 A block diagram depicts further steps of a method for implementing an updatable collaborative 3D map data fusion platform and its persistent virtual world system according to one embodiment;

[0044] Figure 9 A block diagram of a method according to one embodiment is depicted, by which image data is received from first and second client devices, enabling an updatable collaborative 3D map data fusion platform and its persistent virtual world system.

[0045] Figure 10 A block diagram of a method according to one embodiment is depicted, illustrating further steps in a method for implementing an updatable collaborative 3D map data fusion platform and its persistent virtual world system;

[0046] Figure 11 A block diagram according to one embodiment is depicted, illustrating further steps in a method for implementing an updatable collaborative 3D map data fusion platform and its persistent virtual world system. Detailed Implementation

[0047] In the following description, reference is made to the accompanying drawings, which illustrate various embodiments by way of illustration. Furthermore, various embodiments will be described below with reference to several examples. It should be understood that embodiments may include changes in design and structure without departing from the scope of the claimed subject matter.

[0048] Systems and methods for implementing an updatable collaborative 3D map data fusion platform and its persistent virtual world system address one or more of the shortcomings described in the background art. The system and method can employ computer vision techniques to capture and analyze images of at least one coordinate space captured by multiple imaging devices. Features can be extracted and associated with one or more objects within the coordinate space through precise localization and tracking of client devices of the imaging devices, including capturing data and performing a synchronized localization and mapping (SLAM) process, and through posterior analysis techniques. Considering the basic satellite map available in the 3D map data fusion platform, the analyzed feature data can be compared with a Real-Time Virtual Replica Network (RVRN), which includes multiple virtual replicas located within the coordinate space published in the 3D map data fusion platform. This comparison is performed to determine whether the imaging devices should continue capturing data and performing the SLAM process. If further map data is needed because the RVRN does not include mappable features, or because no virtual replicas of the objects are available, the system and method can complete the RVRN map, and then the newly created map can be merged with the published RVRN.

[0049] Following a similar logic, image data from other client devices can be additionally compared with the merged 3D map, and merged with it when not already included, to create a collaborative 3D map that can accept image data input from multiple sources. This collaborative 3D map can be further shared with a persistent virtual world system that includes virtual objects, including virtual copies, pure virtual objects, and applications. Data shared by the collaborative 3D map can be added to the persistent virtual world system as part of the virtual copies. Furthermore, other data and models can be added to the virtual copies for use in multiple applications.

[0050] Figure 1 A schematic diagram of a system 100 according to one embodiment is depicted, which implements an updatable collaborative 3D map data fusion platform connected to a persistent virtual world system.

[0051] System 100 includes a plurality of client devices 102 that capture data 104 from the real world and transmit the data 104 to at least one server computer 106 via network 108. Figure 1Several examples of client devices 102 are provided, such as surveillance camera 102a, laptop computer 102b, robot 102c, drone 102d, smart glasses 102e, autonomous vehicle 102f, smart contact lens 102g, and mobile phone 102h. Of course, other types of client devices 102 can also be used to provide data 104 to at least one server computer 106. Data 104 includes image data 110 containing at least one coordinate space (e.g., one or more locations) with a first plurality of features, and attitude data 112 (e.g., position and orientation) of the client device 102 that captured the image data 110. For example, a first client device (e.g., drone 102d) may capture image data 110 including at least one first coordinate space with a first plurality of features, while a second client device (e.g., mobile phone 102h) may capture image data 110 including at least one second coordinate space with a second plurality of features. Data 104 received by at least one server computer 106 is sent to a 3D map data fusion platform 114 stored in memory, which is connected to a persistent virtual world system 120.

[0052] System 100 receives data from a basic satellite map 116, which can be obtained using map-making techniques (e.g., via various application programming interfaces (APIs)) employing satellite imagery of one or more public or private global, regional, or local maps connected to a 3D map data fusion platform 114. The basic satellite map 116 refers to a standard version of a map, e.g., a map that does not include a dense number of features and details within a region, and some details of these features may be omitted. In some embodiments, the basic satellite map is a 2D map. In other embodiments, the basic satellite map is a basic 3D map. For example, the basic map might be provided by OpenStreetMap. TM Google Maps TM or Crowdmap TM The provided map.

[0053] The persistent virtual world system 120 receives data belonging to a real-time 3D virtual copy network (RVRN 118). RVRN 118 may include multiple virtual copies of real-world objects. Virtual copies may be published by, for example, building owners or national institutions. In some embodiments, published virtual copies are created using computer-aided drafting (CAD) modeling techniques or scanned using one or more computer vision scanning methods. For example, the memory of at least one server computer 106 may include a virtual copy editor, which may include software and hardware configured to enable users to model and edit virtual copies of real-world entities. This virtual copy editor may be CAD software capable of storing data and instructions required for inputting and editing virtual copies. The virtual copy editor may allow input of explicit data and instructions associated with each virtual copy, such as data and instructions describing the shape, location, position and orientation, physical properties, and expected functions and effects of each copy and the system as a whole. Modeling techniques used to convert real-world entities into virtual copies with explicit data and instructions and make them available in the persistent virtual world system may be based on readily available CAD models of the real-world entities. Computer vision scanning methods may include, for example, using synthetic aperture radar, real aperture radar, light detection and ranging (LIDAR), inverse aperture radar, monopulse radar, and other imaging techniques that can be used to map and model real-world entities before integrating them into a persistent virtual world system.

[0054] The 3D map data fusion platform 114 uses data 104 sent from the client device 102 to continue initiating a synchronized localization and mapping (SLAM) process in a first coordinate space. This SLAM process is performed by using the pose of the first client device or data from the base satellite map 108, or a combination thereof, to determine the pose of a first plurality of features within the first coordinate space retrieved from the image data 110. As used herein, the term "pose" refers to the orientation of a device or feature, alone or in combination with other data, such as the position of the device or feature.

[0055] Once sufficient features in the first coordinate space have been used to create a map, the 3D map data fusion platform 114 compares one or more RVRNs 118 in the coordinate space with features extracted from the newly input image data 110 using a feature matching algorithm. If the published RVRNs 118 do not include features and corresponding feature pose data from the image data 110, the 3D map data fusion platform 114 continues to create a first new map with multiple features for the pose of the first client device, and then continues to merge the first new map, which includes at least one three-dimensional coordinate with multiple features, and the RVRNs 118 aligned with the base map to create a fused 3D map.

[0056] A similar process can be applied to image data 110 from other client devices that capture additional features. Thus, for example, the 3D map data fusion platform 114 receives pose data 112 from a second client device and then continues to perform a SLAM process in a second coordinate space using said pose data to determine the pose with a second set of features within the second coordinate space. Once sufficient features in the second coordinate space have been mapped, the 3D map data fusion platform 114 then compares the fused 3D map with the newly input image data using a feature matching algorithm. If the fused 3D map does not include image data 110 containing three-dimensional coordinates, the 3D map data fusion platform 114 continues the SLAM process until a second new map with a second set of features is created. Finally, the 3D map data fusion platform 114 continues to merge the second new map with the fused 3D map to create a collaborative 3D map.

[0057] In this disclosure, in the context of coordinate space, the term "feature" or "image feature" refers to a point of interest within coordinate space that provides information about the content of an image. These features may include points, line segments, and / or surfaces described by any suitable means (e.g., Bézier curves or algebraic varieties) that can describe objects in an environment and can be associated with an image or derived feature. For example, multiple features may be extracted from one or more images of streets, public spaces (e.g., parks or public squares), terrain, and buildings. When these features are combined, a map of the coordinate space including multiple objects in the image can be formed. In other embodiments, features may also include indoor points of interest, such as objects within an indoor location. Features play a fundamental role in many applications of image analysis and processing performed by the 3D map data fusion platform 114, such as applications for image or map data recognition, matching, reconstruction, etc.

[0058] In some embodiments, image data 110 may be included in a frame stream transmitted by one or more client devices 102. The movement and pose of the client device 102, with multiple features, can be determined using images recorded by one or more imaging devices of the client device 102 and SLAM technology performed at least partially on the image data 110 included in the frames by the 3D map data fusion platform 114. In some embodiments, at least some frames may optionally be retained as keyframes during the SLAM process. For example, a new keyframe may be selected from frames showing an incompletely drawn map or an undrawn portion of the map in the coordinate space in question. In some examples, one way to determine that a keyframe shows an incompletely drawn map or an undrawn portion of the map in the coordinate space is when many new features appear, for example, features that do not have matching counterparts in the RVRN 118 or the fused 3D map.

[0059] In some embodiments, the collaborative 3D map can then be shared with a persistent virtual world system 120, which is also stored in the memory of at least one server computer 106. The persistent virtual world system 120 refers to a virtual version of the real world, wherein all or most real-world objects are represented as corresponding virtual copies, and wherein other objects, such as purely virtual objects and applications, can be added and configured. The persistent virtual world system 120 can represent an exact version of the real world. The persistent virtual world system 120 includes real-world coordinates in the form of real-time 3D virtual copies, such as the position, orientation, scale and dimensions, physical properties, and 3D structure of each real-world object.

[0060] In this disclosure, the term "persistent" is used to characterize the state of a system that can continue to exist without continuously executing processes or network connections. For example, the term "persistent" can be used to characterize a virtual world system, in which the virtual world system and all virtual copies, pure virtual objects, and applications contained therein continue to exist after the processes used to create the virtual copies, pure virtual objects, and applications cease, and independently of users connected to the virtual world system. Therefore, the virtual world system can be stored in a non-volatile storage location (e.g., on a server). In this way, virtual copies, pure virtual objects, and applications, when configured to achieve a specific goal, can interact and collaborate with each other even when users are not connected to the server.

[0061] In some embodiments, system 100 may utilize a cloud-to-edge infrastructure that enables distributed computing capabilities using public or private cloud servers, fog servers, and edge devices and systems (e.g., enterprise systems, mobile platforms, and user devices), all connected via a network. Using a cloud-to-edge computing network, access to computing power, computing infrastructure (e.g., through so-called Infrastructure as a Service (IaaS)), applications, and business processes can be delivered to users on demand via client devices as a service. This allows resources, including physical servers and network devices, to enable shared storage and computing, which can be dynamically allocated based on factors such as the distance from the user to the resources and network, and the user's computing needs. In some embodiments, the cloud-to-edge infrastructure also includes a distributed ledger-based infrastructure to facilitate the transmission and storage of data required for the widespread distribution of the 3D map data fusion platform 114 and the persistent virtual world system 120 across shared deployments.

[0062] In some embodiments, to reduce hardware and network requirements, reduce network latency, and improve the general converged reality experience, the system can be connected via a network 108 comprising a combination of millimeter wave (mmW) or mmW and sub-6 GHz communication systems, for example, via fifth-generation wireless system communication (5G). In other embodiments, the system can be connected via a wireless local area network (Wi-Fi) providing 60 GHz data. The provided communication system can allow low latency and high Gbps downlink speeds to field endpoints, meeting the parameters necessary for performing typical highly interactive digital reality applications or other demanding applications. This results in high-quality, low-latency, real-time streaming of digital application content. In other embodiments, the system can communicate via fourth-generation wireless system communication (4G), which may be supported by a 4G communication system, or may include other wired or wireless communication systems.

[0063] In some cellular network embodiments, the mmW band, also known as the extremely high frequency band, can be used. The mmW band spans from 30 to 300 GHz; however, adjacent ultra-high frequencies from approximately 10 to 300 GHz can also be included, as these waves resemble mmW propagation. Due to the extremely high frequencies of mmW, mmW-based antennas, or combinations of mmW-based antennas and sub-GHz antenna systems, are highly reflective, easily blocked by walls or other solid objects, and may experience significant attenuation when passing through foliage or in inclement weather conditions. Therefore, cellular network antennas can include small mmW transceivers arranged in a grid pattern, which helps to amplify total energy, increase gain, and reduce power loss without increasing transmission power. Techniques known in the art (e.g., multiple-input multiple-output or MIMO) can be used to simultaneously split beams on several devices or send multiple data streams to a single device, thereby improving Quality of Service (QOS). Additionally, the antenna can cover a relatively small area between approximately 100 meters and approximately 2 kilometers to ensure accurate propagation of millimeter waves.

[0064] According to one embodiment, tracking of the client device 102 is performed by estimating one or more of the following: time of arrival (TOA), angle of arrival (AOA) / angle of departure (AOD), and visual imaging techniques (e.g., by an imaging device located in a region close to the client device 102). Tracking can be performed by one or more of the following: radar technology, antenna, Wi-Fi, inertial measurement unit, gyroscope, and accelerometer.

[0065] In some embodiments, due to the higher density of mmW antennas per square meter compared to other systems, the availability of antennas closer to the client device is higher, resulting in a shorter arrival time between the signal and the transmitter, and vice versa. This can be used to calculate the estimated location and orientation of the client device 102 via TOA or other techniques (e.g., estimated AOA / AOD). Furthermore, the estimated location of the client device provides necessary information for beamforming design. Therefore, once the location of the client device 102 is accurately estimated, the antenna can be directly beamed to the client device 102 using a line-of-sight (LOS) path, via a first-order reflector, or via any other suitable technique. This results in synergy between localization and communication services while maintaining high QoS. Moreover, due to the proximity of the antenna and the client device 102 and the higher QoS, a larger portion of the SLAM process can be performed at the server 106, enabling the use of lighter or thinner client devices 102.

[0066] In other embodiments, Global Navigation Satellite System (GNSS) refers to any satellite-based navigation system, such as GPS, BDS, GLONASS, QZSS, Galileo, and IRNSS, which can be used to achieve positioning of client device 102. Utilizing signals from a sufficient number of satellites and techniques such as triangulation and trilateration, GNSS can calculate the device's position, velocity, altitude, and time. In one embodiment, the external positioning system is enhanced using Assisted GNSS (AGNSS) via an existing cellular communication network architecture, where the existing architecture includes 5G. In other embodiments, the AGNSS tracking system is further supported by a 4G cellular communication network. In indoor embodiments, GNSS is further enhanced via a wireless local area network (e.g., Wi-Fi), including but not limited to providing data at 60 GHz. In alternative embodiments, GNSS is enhanced via other technologies, such as differential GPS (DGPS), satellite-based augmentation systems (SBASs), and real-time motion (RTK) systems. In some embodiments, tracking of client device 102 is achieved through a combination of AGNSS and inertial sensors in client device 102.

[0067] Figure 2 A schematic diagram of another embodiment of system 200 is shown, which implements an updatable collaborative 3D map data fusion platform and its persistent virtual world system. Figure 2 Some elements can be similar to Figure 1 The elements are such that the same or similar reference numerals can be used to represent them.

[0068] System 200 includes multiple client devices 102, such as client device 1, client device 2, and client device N, all of which are capable of sending data to the client management module 202 of the 3D map data fusion platform 114. Each client device 102 dynamically creates a separate client device management instance 204 to receive and process the data 104 transmitted by each client device 102 and to send corresponding map updates 206 to each client device 102 for further map drawing and navigation. For example, the processing of data 104 by a single client device management instance 204 may include implementing a SLAM pipeline to perform a coordinate space SLAM scan based on the received data 104; implementing one or more feature matching algorithms to compare image data with one or more RVRNs 118 in the persistent virtual world system 120; and creating a coordinate space map based on or aligned with a base satellite map 116, and merging this map with available RVRNs 118 to create a fused 3D map 208. Figure 3 Further details of the tasks performed by the client management module 202 are described in the document.

[0069] In some embodiments, image data is received (e.g., synchronously) from multiple client devices. In these embodiments, the 3D map data fusion platform 114 may be configured to first obtain the pose of client devices 102 (e.g., client devices 1 and 2) and simultaneously perform a SLAM process in first and second coordinate spaces by capturing image data from at least one imaging device of the first and second client devices, wherein the SLAM process is performed by determining poses with first and second plurality of features in the first and second coordinate spaces using the poses of the first and second client devices 102. Then, each individual client device management instance 204 is configured to simultaneously obtain data from the corresponding first or second client device 102 and perform a SLAM process for each corresponding coordinate space. The 3D map data fusion platform 114 may be further configured to check whether sufficient features have been extracted to subsequently compare and match plurality of features in each coordinate space. If the first and second plurality of features match, the 3D map data fusion platform may continue to merge the first and second feature maps, wherein the merging is performed in alignment with a base satellite map 116 to form a fused 3D map 208. The fused 3D map 208 can then be merged with RVRN 118 to generate a collaborative 3D map. If the first and second feature maps do not match, the 3D map data fusion platform 114 can further compare and merge the first and second feature maps with the corresponding RVRN 118 separately to generate a collaborative 3D map.

[0070] In this disclosure, the term "instantiation" refers to the process of virtualizing server attributes (e.g., storage, computing power, network, operating system, etc.), such as one or more cloud server computers 106, in order to create an instance of one or more cloud server computers 106 or its programs, such as an instance of client management module 202.

[0071] Figure 3 A schematic diagram of an embodiment of system 300 is depicted, which implements an updatable collaborative 3D map data fusion platform and its persistent virtual world system. Figure 3 Some elements can be similar to Figure 1-2 The elements are such that the same or similar reference numerals can be used to represent them.

[0072] exist Figure 3 In this system 300, a client device 102 (e.g., client device 1) is included that transmits data 104 to one or more server computers 106. The client device 102 includes a tracking unit 302, which includes a sensor 304 and a transceiver 306. The sensor 304 may be implemented as computing hardware, software, and / or firmware adapted to acquire at least visual data from the real world and determine / track the attitude 112 of the client device 102. In some embodiments, to capture image data, the sensor 304 includes one or more optical sensors, including one or more of an RGB camera, a color camera, a grayscale camera, an infrared camera, a charge-coupled device (CCD), a CMOS device, a depth camera (e.g., LiDAR), or combinations thereof. To determine the attitude of the client device 102, the sensor 304 also includes one or more inertial measurement units (IMUs), accelerometers, and gyroscopes, which can be used to internally determine the attitude of the client device 102 using an inertial navigation system (INS) algorithm.

[0073] Transceiver 306 can be implemented as computing hardware and software, configured to enable the device to receive radio waves from the antenna and transmit data back to the antenna. In some embodiments, an mmW transceiver 306 can be used, which can be configured to receive mmW wave signals from the antenna and transmit data back to the antenna. Transceiver 306 can be, for example, a bidirectional communication transceiver 306.

[0074] Tracking unit 302 can be implemented by combining the capabilities of an IMU, accelerometer, and gyroscope with the position tracking provided by transceiver 306 and the precise tracking, low latency, and high QoS capabilities provided by a mmW-based antenna, thus enabling sub-centimeter or sub-millimeter position and orientation tracking. Accuracy can be increased when tracking the real-time position and orientation of device 102. In an alternative embodiment, sensor 304 and transceiver 306 can be coupled together in a single tracking module device. At least one processor of client device 102 can use algorithms such as Kalman filters or extended Kalman filters to address probabilistic uncertainties in the sensor data during tracking of client device 102. In other embodiments, at least one processor can also perform jitter reduction algorithms during tracking of client device 102, for example, by maintaining a constant and consistent error during the SLAM process, rather than estimating the error at each step of the process.

[0075] Data 104, including pose 112 and image data 110, can be transmitted to a 3D map data fusion platform 114 to perform one or more operations on the data 104. In some embodiments, reference... Figure 2 At least some operations are performed by a single dedicated instance of the client management module 202 generated on demand.

[0076] The 3D map data fusion platform 114 includes multiple modules dedicated to one or more specific functions. These modules can communicate with or with the persistent virtual world system 120 via corresponding APIs. In this disclosure, "module" can refer to a dedicated computer circuit, software application, firmware, a set of computer instructions / software running on a processor, or the processor itself (e.g., an ASIC) for performing a specific function. Some modules of the 3D map data fusion platform 114 include a basic satellite map 308, a SLAM pipeline 310, which includes multiple modules running computational processes (e.g., commands, program execution, tasks, threads, procedures, etc.), such as a new feature extraction module 312, a feature pose inference module 314, a new map generation module 316, and a map comparison and merging module 318; and other modules, such as a fused 3D map 320 and a collaborative 3D map 322. Depending on the implemented process, modules may sometimes implement processes sequentially or in parallel and may receive feedback from each other, which can be used for further processing.

[0077] During the coordinated implementation of the SLAM pipeline 310, image data 110 is captured by one or more imaging devices of the client device 102. In some embodiments, the coordinated implementation of the SLAM pipeline 310 is performed at least in part by the 3D map data fusion platform 114 and the client device 102 via a SLAM scanning and navigation module 334 stored in the memory of the client device 102 and executed by at least one of its processors. In some embodiments, the image data 110 may be preprocessed by computer instructions from the SLAM scanning and navigation module 334 executed by at least one processor of the client device 102. For example, at least one processor may generate abstract image data based on the captured image data 110 before the image data is sent to the server 106. In other examples, in embodiments where the client device 102 is a wearable device (e.g., a head-mounted display), the SLAM scanning and navigation module 334 of the client device 102 may be used to translate the movements of a user's body or head while wearing the client device 102, which may be necessary for performing SLAM.

[0078] The new feature extraction module 312 of the SLAM pipeline 310 can be configured to extract features from image data 110 by extracting visual and structural information about the environment surrounding the client device from frames containing image data 110. For example, the new feature extraction module 312 can determine the geometry and / or appearance of the environment, for instance, based on analysis of preprocessed sensor data received from the SLAM scanning and navigation module 334 of the client device 102. Feature determination or extraction can be performed using several algorithms, such as corner and blob detection, Harris detection, Harris-Laplace detection, Shi-Tomasi, Scale Invariant Feature Transform (SIFT), Accelerated Robust Features (SURF), Features from Accelerated Segmentation Test (FAST), Binary Robust Independent Basic Features (BRIEF), or ORB feature detection algorithms or combinations thereof. In some embodiments, the new feature extraction module 312 can further perform a feature tracking and matching process. Feature tracking is the process of finding correspondences between features in adjacent frames. For example, feature tracking can be useful when small motion changes occur between two frames. Feature matching is the process of extracting features individually and matching them across multiple frames, which can be useful when the appearance of features changes significantly. In some embodiments, the new feature extraction module 312 may further perform outlier identification and suppression using any suitable technique, such as the RANSAC method and / or matching equality techniques.

[0079] Subsequently, the feature pose inference module 314 of the SLAM pipeline 310 can infer the pose of the extracted features relative to a given pose of the client device 102. In some examples, this calculation can be performed using methods such as Harris corner detection, Canny edge detection, known feature-image interrelationships, region discovery via color clustering, and boundary discovery via gradient aggregation. The pose of each feature can be determined by comparing the feature's positional data with other geometric data. Then, the new map generation module 316 of the SLAM pipeline 310 can create a new feature map including the corresponding poses. In some cases, the map may not include a large number of features, which may require the SLAM pipeline 310 to continue until a sufficient number of features are reached. During the new map generation module 316, the SLAM pipeline 310 can simultaneously compare and match the features of the new map with those included in the RVRN 118, which can be performed by the map comparison and merging module 318. Therefore, the RVRN 118, including multiple virtual copies, can be considered in these embodiments as a detailed 3D map with multiple features including corresponding objects. If RVRN 118 does not include map-drawing features or only partially includes map-drawing features, the map comparison and merging module 318 can continue to merge the features of the newly generated map with RVRN 118 relative to the base satellite map 308 to generate a merged 3D map 320.

[0080] In some embodiments, the second or more client devices may provide data 104 to the server 106 synchronously or asynchronously. In these embodiments, during the coordinated implementation of the SLAM pipeline 310, image data 110 is captured by one or more imaging devices of the second client devices. A similar process as described above can be implemented, such that a SLAM process in a second coordinate space is initiated using the data 104 captured by the second client devices. Once a map is drawn for a sufficient number of features in the second coordinate space, the SLAM pipeline 310 continues to compare a second plurality of features of the drawn map with the fused 3D map 320 using a feature matching algorithm. If the fused 3D map 320 does not include features of the newly drawn map, the SLAM pipeline 310 may continue the SLAM process until a second map is created, which is then merged with the fused 3D map 320 by the collaborative 3D map data fusion platform 114 to create a collaborative 3D map 322.

[0081] The 3D map data fusion platform 114 can continue to share the collaborative 3D map 322 with the persistent virtual world system 120, which can communicate via a corresponding API. Therefore, the collaborative 3D map 322 is then included in the form of RVRN 118 as part of the persistent virtual world system 120. The persistent virtual world system 120 may also include other virtual objects 324, such as applications 326 and purely virtual objects 328 (e.g., virtual objects that do not exist in the real world and therefore do not have real-world counterparts, unlike RVRN 118).

[0082] In some embodiments, a client device 102 connected to at least one server computer 106 enables a user to select one or more virtual objects 324 of a persistent virtual world system 120 and overlay the selected virtual objects 324 onto one or more locations using a collaborative 3D map 322 shared with the persistent virtual world system 120 as a reference for alignment. For example, a user can overlay a purely virtual object 328 or an application 326 onto a collaboratively created 3D scene and view the overlaid object in augmented reality (AR) or virtual reality (VR) via an AR / VR module 330 stored in the memory of the client device 102.

[0083] In some embodiments, the generated collaborative 3D map 322 also includes rights information for each user that can be added to the processed image data by the 3D map data fusion platform 114, wherein only authorized users have the right to view the image data for that location. For example, some restricted indoor areas may only be visible to certain people whose permission level allows them to see these areas, making them inaccessible to “normal” users.

[0084] In some embodiments, application 326 may be one or more conventional applications, distributed applications, or decentralized applications. Conventional applications are typically based on a traditional client-server model and run on dedicated servers in a static infrastructure. Distributed applications are applications primarily stored on cloud computing platforms, such as the cloud servers disclosed herein, and can run simultaneously on multiple systems and devices on the same network, or can run on a blockchain or a distributed database based on a distributed ledger. Decentralized applications primarily run on decentralized infrastructure, such as a blockchain or a distributed database based on a distributed ledger. The interaction mechanism with application 326 is defined by computer code included in computer scripts and computer programs, and implemented through application 326, smart contracts available in the blockchain or the distributed database based on a distributed ledger, or a combination thereof. Users can experience interaction through game-like interactive applications or game-like interaction mechanisms.

[0085] In some embodiments, server 106 may be further configured to add logic, virtual data, and models to at least some virtual objects in a persistent virtual world system to provide self-computation capabilities and autonomous behavior. Models may include, for example, one or more of 3D models, dynamic models, geometric models, or machine learning models, or combinations thereof. Therefore, this disclosure implements a way of combining a collaborative SLAM 3D mapping process with a persistent virtual world system 120, which may include high-level features beyond the geometric and visual attributes of structures in a defined coordinate space for navigation purposes, but may also enable interaction with said structures (e.g., in the form of virtual objects 324) and communication between at least some structures. For example, a collaborative 3D map shared with the persistent virtual world system 120 of this disclosure can be used for efficient autonomous vehicle traffic management because vehicles can travel on roads included in the collaboratively mapped persistent virtual world system 120, and virtual copies of vehicles include logic, virtual data, and models for autonomous behavior such as stopping or adjusting traffic speed during accidents, or recommending / manipulating vehicles to travel on the most efficient routes during traffic jams. In other examples, collaborative 3D maps shared with the persistent virtual world system 120 disclosed herein can be used for the management of manufacturing operations, utilities, pollution control, etc.

[0086] "Self-computing capability," also known as "self-management capability," refers in this paper to the ability of a software entity (e.g., a virtual copy of the persistent virtual world system 120 or RVRN 118) to autonomously perform tasks using artificial intelligence algorithms, such as managing distributed computing resources and adapting to changes in its environment. Self-management rules and conditions can be further managed using smart contracts running on a blockchain or based on distributed ledger technology, further codifying rules and conditions in a distributed and transparent manner. Therefore, each virtual copy can act autonomously based on conditions in the real world reflected in the persistent virtual world system 120, by allocating necessary resources, autonomously sending and executing commands, and generating events according to the needs of each environment. Achieving this type of behavior may require training the virtual copy with artificial intelligence algorithms during the modeling of the virtual copy.

[0087] Figure 4 A schematic diagram depicts a SLAM scan performed by two or more client devices and the data flow to the server according to one embodiment.

[0088] exist Figure 4 In this process, user 402 wears a first client device (e.g., wearable device 404), which captures a first image dataset 406 from a first coordinate space 408 via a SLAM process. This SLAM process is, for example, a coordinated SLAM process implemented by a SLAM scanning and navigation module 334 implemented in wearable device 404 and a SLAM pipeline 310 implemented in server computer 106. Figure 3 As shown. Wearable device 404 can also send first client attitude data 410 and first client device personal ID code 412 to at least one server computer 106. Similarly, a second client device (e.g., drone 414) captures a second image dataset 416 in a second coordinate space 418 and sends the second image dataset 416 along with corresponding second client device attitude data 420 and personal ID code 422 to at least one server computer 106. Communication between the client devices and at least one server computer 106 can be achieved via one or more antennas 424, such as mmW antennas (e.g., antenna communication systems for 5G). Personal ID codes 412 and 422 can be, for example, QR codes, URLs, IP addresses, MAC addresses, cryptographic hashes, universally unique identifiers, or organization-unique identifiers. ID codes 412 and 422 are linked to the 3D coordinates of the total area mapped by each client device.

[0089] In the first phase, the 3D map data fusion platform 114 receives data from, for example, a wearable device 404 and compares the data with available RVRNs 118 in the persistent virtual world system 120 via a map comparison and merging module 318. In some embodiments, the 3D map data fusion platform 114 references... Figure 3 The corresponding modules 312-316 described perform new feature extraction (e.g., a first plurality of features extracted from the first image dataset 406), feature pose inference, and new map generation. When sufficient data is available for comparison, and if the comparison results in the available RVRN 118 not including or not fully including the first plurality of features, the 3D map data fusion platform 114 continues the SLAM process and creates a client-identified first map 426, which refers to the first coordinate space 408 of the SLAM-drawn map including the first client device personal ID code 412. The client-identified first map 426 can be further compared and combined with existing data in the persistent virtual world system 120 to form a fused 3D map 320 including the client-identified region.

[0090] Data transmitted by the drone 414 can also be received by the 3D map data fusion platform 114, and can be used for feature extraction, feature pose inference, and new map generation through corresponding modules 312-316, as shown in the reference. Figure 3 As described above, when sufficient data is available for comparison, and if the comparison results in the available RVRN 118 do not include or do not fully include the first plurality of features, the 3D map data fusion platform proceeds to 114 to continue the SLAM process and create a client-identified second map 428, which refers to the second coordinate space 418 of the SLAM-drawn map including the second client device personal ID code 422. The client-identified second map 428 is then compared with the fused 3D map 320 by the map comparison and merging module 318, and combined if necessary, to form a collaborative 3D map that is subsequently shared with the persistent virtual world system 120.

[0091] In some embodiments, the map area contribution identified by the client is further stored in a distributed ledger. In yet other embodiments, the map area contribution identified by the client can be used in a reward system that includes providing rewards associated with the map area contribution of the client device. For example, the coordinates and feature points in these coordinates, as well as the pose data of each feature point, can be stored in ledgers or blocks, which are growing lists of records formed by cryptographic linking and protection to create a blockchain. These blockchains can be designed to resist data modification and can serve as open distributed ledgers that can record the areas mapped by each client device to support the reward system. Each block can contain a hash pointer as a link to the previous block, a timestamp, transaction data, and other data associated with the area mapped by the client device. For example, if the area mapped by the first client device is twice the size of that of the second client device, the reward system can calculate twice the reward (e.g., cryptocurrency, coupons, etc.) for the first client device. These transactions can also be stored in the distributed ledger in a verifiable and permanent manner. The rules of such a reward system can be managed by smart contracts stored and implemented in the distributed ledger.

[0092] Figure 5 A diagram illustrating the components of an RVRN according to one embodiment is shown. Figure 5 Some elements can refer to Figure 1-4 Similar or identical elements can therefore use the same reference numerals.

[0093] like Figure 5 As shown, elements in the real world 502, including other devices 504 and user devices 506, provide data streams 508 to the persistent virtual world system. These data streams 508 can be unidirectional or bidirectional, depending on the capabilities of the other devices 504 or the user devices. In this disclosure, the term "user device" can refer to a device used by a human user to interact with the application, such as a mobile device, personal computer, game console, media center, and head-mounted display. In this disclosure, other devices 504 refer to technology-based systems that can connect and share data to networks other than user devices 506, such as surveillance cameras, vehicles, traffic lights, buildings, streets, train tracks, home appliances, robots, drones, etc.

[0094] The data stream 508 transmitted by other device 504 may be obtained by sensors mounted on other device 504, such as one or more optical sensors, temperature sensors, proximity sensors, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, and magnetometers), infrared sensors, pollution sensors (e.g., gas sensors), pressure sensors, light sensors, ultrasonic sensors, smoke sensors, touch sensors, color sensors, humidity sensors, water sensors, electrical sensors, or combinations thereof. For example, data stream 508 may include image data and attitude data, which may be sent to a 3D map data fusion platform for further processing before integrating feature data into the virtual copy 510. In addition to sensor data, data stream 508 of user device 506 may include user input data generated by interaction with applications via user device 506.

[0095] By providing a sensor mechanism that continuously captures data from the real world 502 to multiple other devices 504 and user devices 506, the virtual world and each synchronized virtual copy 510 are kept updated with real-time multi-source data reflecting the conditions of the real world 502. The multi-source data includes captureable data for each real-world element, including one or more of the following: 3D image data, 3D geometry, 3D entities, 3D sensory data, 3D dynamic objects, video data, audio data, priority data, chemical composition, waste generation data, text data, time data, location data, orientation data, speed data, temperature data, humidity data, pollution data, lighting data, volume data, flow data, colorimetric data, power consumption data, bandwidth data, and quality data.

[0096] Interaction with virtual replicas 510 in the persistent virtual world system is achieved through data exchange using a publish / subscribe service connected to a data storage 512 of each virtual replica 510. Data types can include periodic and aperiodic, synchronous and asynchronous data. Each virtual replica 510 maintains a data storage 512 where data can be updated by real-world counterparts or microservices (not shown). Data in the persistent virtual world system can be directly associated with a specific virtual replica 510, or it can be processed as anonymous data, which can include aggregations of multiple streaming sources from related virtual replicas. For example, data from all units of a specific model of a car can be processed and aggregated into a data stream for predictive maintenance services.

[0097] Microservices are individual services that can be developed, deployed, and maintained independently. Each service is configured to perform discrete tasks and communicate with other services via APIs. Each microservice can use a virtual replica model and its relationship to the environment to update data in the data storage 512 of the virtual replica 510 to modify specific attribute values ​​of the virtual replica 510. Microservices can use specific core services of a persistent virtual world system, such as multiple engines, or connect to external platforms.

[0098] Data stream 508 can be stored in data storage 512 via connector 514. Connector 514 may include software, hardware, and / or firmware for reading data from data stream 508 and writing data to data storage 512. Connector 514 may use a publish / subscribe application programming interface (API) to connect to data storage 512 and facilitate feeding data stream 508 from other devices 504 and user device 506 to virtual copy 510. Data stream 508 from other devices 504 is further fed to cyber-physical system 516 via connector 514, while data stream 508 from user device 506 is fed to virtual avatar 518 of user 506 via connector 514. System 500 also includes an implementation of robot 520, which may include software, hardware, and / or firmware configured to respond in an automated agent manner using machine learning algorithms to exhibit human or human-like behavior. Human avatar 518 may be configured to display the physical characteristics of a human user or may be configured to have different visual aspects and features.

[0099] In this disclosure, a virtual-real twin or twin pair can be considered a cyber-physical system 516, which is an integration of computation and physical processes, the behavior of which is defined by the network and physical components of the system. Therefore, the virtual copy 510 is the network component of the cyber-physical system 516. The virtual copy 510 can then be considered an extension of the real twin pair, allowing the physical components to be coupled with artificial intelligence and simulation to enhance the capabilities and performance of the object. In some embodiments, the virtual copy 510 can be a substitute for a portion of the physical components and processes. For example, in the event of a sensor failure in its real counterpart, the sensing input of the real twin pair is provided by the interaction of the virtual twin pair in the virtual world. In another example, if the real twin's battery is low, some computation of the real twin pair can be performed in the virtual world.

[0100] The virtual copy 510 may also include a model 522, which refers to any graphical, mathematical, or logical representation of aspects of reality that can be used to replicate reality in a persistent virtual world system. In some embodiments, suitable models 522 include one or more of a 3D model 524, a geometric model 526, a dynamic model 528, and a machine learning model 530. Although only four models 522 are disclosed herein, those skilled in the art will understand that the system can be adapted to implement fewer or more models than those presented.

[0101] 3D model 524 is used in conjunction with geometric model 526 to illustrate data included in each geometry of virtual copy 510, such as textures, colors, shadows, reflections, collision effects, etc. 3D model 524 includes a 3D data structure for visually representing virtual copy 510 and other virtual elements in the persistent virtual world system, such as applications, purely virtual copies, virtual robots, advertisements, etc. The 3D data structure may include, for example, one or more octrees, quadtrees, BSP trees, sparse voxel octrees, 3D arrays, kD trees, point clouds, wireframes, boundary representations (B-Rep), constructive solid geometry trees (CSG trees), binary trees, and hexagonal structures. The 3D data structure has the capability to accurately and efficiently represent data for each geometry of virtual objects in the persistent virtual world system. The correct choice of 3D data structure depends on: the source of the data; the geometric accuracy sought during rendering; whether rendering is done in real time or pre-rendered; whether rendering is performed via a cloud server, via a user device, a fog device, or a combination thereof; the specific application using a persistent virtual world system, for example, medical or scientific applications may require a higher level of clarity compared to other types of applications; the memory capacity from the server and user device and the therefore expected memory consumption; and others.

[0102] The geometric model 524 includes a mathematical model that defines the shape of the virtual copy 510 based on real-world elements, and can be supplemented by the 3D model 524.

[0103] Dynamic model 528 represents a mathematical model that describes the behavior of real-world objects in a virtual world over time. It may include a set of states that appear in a defined order and may include continuous (e.g., algebraic or differential equations) and discrete (e.g., as a state machine or stochastic model) dynamic models.

[0104] Machine learning model 530 is a mathematical representation of real-world objects and processes, typically generated by a machine learning algorithm based on actual or simulated data that has already been used as training data for learning. This model can implement artificial intelligence techniques that can be used to optimize the operation and / or performance of a real twin pair through a virtual twin pair. The machine learning model can employ machine learning algorithms that allow the virtual copy 510 to be taught about the behavior of the real twin pair in order to simulate the behavior of the real twin pair.

[0105] In some embodiments, the model used in the virtual copy 510 takes into account the level of detail (LOD) required for scene-specific computations. LOD includes reducing the complexity of the representation of model 522 as the virtual copy 510 moves further away from the viewer, or based on other metrics such as object importance, relative viewpoint velocity, viewer classification, or position. LOD is a feature commonly used in game engines to optimize real-time rendering, which uses a more detailed model only where the user's viewpoint is closer to the object. LOD management improves the efficiency of computational processes (e.g., rendering processes) by reducing the workload of the graphics pipeline (typically vertex transformations) or by enhancing physical simulations, as different physical models can be associated with the virtual copy in order from low-fidelity to high-fidelity models, enabling different simulations to be performed depending on the situation and circumstances. LOD management improves the efficiency of computational processes (e.g., rendering processes) by reducing the workload used by the graphics pipeline (typically vertex transformations) or by enhancing physical simulations, because different 3D models 524 or dynamic models 528 can be associated with virtual copies 510 in order from low-fidelity models to high-fidelity models, allowing for different simulations depending on the situation and circumstances. Generally, LOD management can improve frame rates and reduce memory and computational requirements.

[0106] Multiple connected virtual copies 510 form an RVRN 118. Each virtual copy 510 can also display social connections 532 with each other, i.e., interactions between them.

[0107] In some embodiments, the virtual copy 510 includes one or more of the following: 3D world and building data, such as SLAM or derived map data, 3D geometric data, 3D point cloud data, or geographic information system data representing the structural properties of the real world, which can be used to model 3D structures for digital reality applications.

[0108] In some embodiments, each virtual copy 510 may be geolocated using a reference coordinate system suitable for the current geolocation technology. For example, the virtual copy may use a World Geodetic System standard, such as WGS84, which is the current reference coordinate system used by GPS.

[0109] Figure 6 A diagram of a system 600 according to one embodiment is depicted, which describes the platform and interface for generating RVRNs used in a merged reality system. Elements within dashed lines represent virtual copies and the persistent virtual world system 120 in which RVRN 118 resides.

[0110] like Figure 6 As shown, RVRN 118 can connect to multiple external platforms 602 or to engine services 604 included in the persistent virtual world system 120. These external platforms may include, for example, Internet of Things (IoT) platforms, machine learning (ML) platforms, big data platforms, and simulation platforms, which can connect to the persistent virtual world system 120 via API 606 to provide and manipulate models and consume or publish data to the virtual replica.

[0111] An IoT platform refers to the software, hardware, and / or firmware capable of managing multi-source input data received from sensors in other devices and user devices. An ML platform refers to the software, hardware, and / or firmware that provides the RVRN 118 with the ability to use machine learning models and algorithms for artificial intelligence applications. A big data platform refers to the software, hardware, and / or firmware that enables organizations to develop, deploy, operate, and manage big data related to the RVRN 118. A simulation platform refers to the software, hardware, and / or firmware that enables the use of the RVRN 118 and its data and models to virtually reproduce the real-world behavior of entities.

[0112] The engine service 604 included in the persistent virtual world system 120 may include, for example, an artificial intelligence engine, a simulation engine, a 3D engine, and a haptic engine. An artificial intelligence engine refers to software, hardware, and / or firmware capable of managing and applying machine learning models and algorithms for artificial intelligence applications. A simulation engine refers to software, hardware, and / or firmware capable of using virtual copies and their data and models to virtually reconstruct the realistic behavior of real-world entities. A 3D engine refers to software, hardware, and / or firmware that can be used to create and process 3D graphics for virtual copies. A haptic engine refers to software, hardware, and / or firmware capable of implementing haptic features on applications and virtual copies to provide touch-based interaction to the user. The persistent virtual world system 120 is also connected to a spatial data streaming platform configured for the optimized exchange and management of real and virtual spatial data within the persistent virtual world system 120 and between the persistent virtual world system 120 and merged reality 608.

[0113] Some engine services 604 (e.g., 3D engines and haptic engines) can connect to merged reality 608 via a suitable digital reality interface 610 in the user device, enabling access to merged reality in either virtual reality or augmented reality. Merged reality 610 provides the user with an extended reality in which real elements are overlaid or enhanced by persistent virtual objects, anchored to real elements in a specific geographic location or reality, and includes simulations of AI and virtual copies connected to the reality. The user can interact with merged reality 608 without restriction through his / her avatar.

[0114] In some embodiments, the 3D map data fusion platform 114 can acquire and process collaboratively drawn map data in order to compare, match, and combine several detailed 3D maps with the existing RVRN 118 of the persistent virtual world system 120.

[0115] RVRN 118 is a component of persistent virtual world system 120 and enables virtual copy reality 612, in which all real-world elements are entirely virtual and can be virtually enhanced (e.g., adding features to the virtual copy that real-world elements might not possess). In this disclosure, virtual copy reality 612 differs from the typical concept of virtual reality, which can represent an immersive realization of a world where all elements are virtual, while virtual copy reality 612 takes into account context, precise geolocation based on real-world objects, and interactions and connections between virtual copies. Virtual copy reality 612 can be continuously updated (or periodically or intermittently updated as computing resources and conditions permit) through data and models and can be manipulated through multiple platforms and / or engines. Therefore, virtual copy reality 612 refers to an actual virtual copy of a world within a persistent virtual world system, where the persistent virtual world system provides data, models, interactions, connections, and infrastructure enabling each virtual copy to possess self-computing capabilities and autonomous behavior.

[0116] In some other embodiments, system 600 may store a separation layer for augmented reality and virtual reality in the memory of at least one server. The separation layer may allow access to any virtual copy reality 612 in either augmented reality or virtual reality via merged reality 608, and may be activated by a user device connected to at least one server computer whenever one or the other type of reality is accessed. Each layer may include augmentations specific to each layer's reality and virtual copy. For example, when accessing merged reality 608 in augmented reality, a user may view real objects located in the current merged reality scene, the current augmentations of each real object via the corresponding virtual copy, and purely virtual objects configured to be visible only in augmented reality. In another example, when viewing merged reality 608 in virtual reality, the user may only view the version of virtual copy reality 612 configured for virtual reality, including augmentations configured only for the virtual reality view. However, when in virtual reality, the user may activate the augmented reality layer to view the augmentations and virtual objects originally specified for augmented reality. Similarly, when in augmented reality, the user may activate the virtual reality layer to be fully transported to virtual reality while still being able to view the augmentations in augmented reality.

[0117] In a further embodiment, RVRN 118 is connected to a distributed ledger platform 614, which can store client-identified map area contributions 616. In some embodiments, client-identified map area contributions 616 can be used in a reward system that includes rewards provided by the client device in relation to the map area contributions. These rewards can be managed by a corresponding smart contract 618 to provide users with appropriate rewards (e.g., cryptocurrency 620) based on their map contributions.

[0118] Figure 7 A method 700 is described, according to one embodiment, as a block diagram of an updatable collaborative 3D map data fusion platform and its persistent virtual world system.

[0119] The method 700 of this disclosure begins in steps 702 and 704, providing a 3D map data fusion platform (e.g., reference) in the memory of a server computer including at least one processor. Figure 1-4 (as described in section 6, the 3D map data fusion platform 114). In step 706, method 700 continues, providing a basic satellite map of the world location in the 3D map data fusion platform. In some embodiments, the basic satellite map is obtained using satellite mapping techniques (e.g., via multiple corresponding APIs) that utilize satellite imagery of one or more public or private global, regional, or local maps connected to the 3D map data fusion platform.

[0120] In step 708, the method continues, publishing the RVRN (e.g., from a base satellite map, if available) of the real-world location in the 3D map data fusion platform, referencing a base satellite map (e.g., aligned with the coordinates of the base satellite map). Figure 1-6 (RVRN 118). Following this process, in step 710, method 700 continues to obtain the attitude of the first client device, which can be performed using position and orientation determination and tracking techniques and algorithms from sensor data obtained from sensors (e.g., IMU, accelerometer, and gyroscope) by using external sensors (e.g., cameras or combinations thereof). Subsequently, the method continues in step 712 to perform a SLAM process in a first coordinate space, wherein the SLAM process uses image data from at least one imaging device of the first client device and the attitude of the first client device to determine an attitude with a first plurality of features in the first coordinate space. In some embodiments, during the coordinated implementation of the SLAM pipeline, image data is captured by one or more imaging devices of the client device, which is at least partially performed by the 3D map data fusion platform and the client device through a SLAM scanning and navigation module stored in the memory of the client device and executed by at least one of its processors. In some embodiments, the image data can be preprocessed by computer instructions of the SLAM scanning and navigation module executed by at least one processor of the client device.

[0121] Method 700 continues, creating a new map containing a first set of features located in the first coordinate space. Figure 7 In the example shown, this includes performing the function in step 714: checking whether enough features have been extracted from the image data to compare the new data with the RVRN data. If not enough features have been captured, the method continues in step 712: the SLAM process continues. Once a map has been drawn with enough features in the first coordinate space, method 700 continues in step 716, using a feature matching algorithm via a 3D map data fusion platform to compare the first plurality of features of the drawn map with the published RVRN to check whether the first plurality of features of the drawn map are included or partially included in the RVRN. If the first plurality of features of the drawn map are not included or are only partially included in the RVRN, the SLAM process can continue in step 718 until a first new map is created. Then, as shown in step 720, the first new map can be merged with the RVRN to create a merged 3D map. However, if the published detailed RVRN includes features of the newly drawn map, method 700 can terminate in terminator 722.

[0122] Figure 8A block diagram of a method 800, according to one embodiment, is depicted that can be used in conjunction with a method 700 for implementing an updatable collaborative 3D map data fusion platform and its persistent virtual world system.

[0123] Method 800 begins with steps 802 and 804, obtaining the position and orientation of the second client device. Then, in step 806, the method continues by performing a SLAM process in a second coordinate space, which uses image data from at least one imaging device of the second client device and the pose of the second client device to determine a pose with a second plurality of features in the second coordinate space. In step 808, method 800 continues by checking whether sufficient features have been extracted from the image data to compare the new data with the fused 3D map data.

[0124] If not enough features are captured, the method continues in step 806, continuing the SLAM process. Once a map has been drawn with enough features in the first coordinate space, method 800 continues, creating a second new map that includes a second set of features. Figure 8 In the example shown, this includes performing the function in step 810, comparing a second plurality of features of the drawn map with the fused 3D map using a feature matching algorithm via a 3D map data fusion platform to check whether the second plurality of features of the drawn map are included or partially included in the fused 3D map. If the first plurality of features of the drawn map are not included or are only partially included in the fused 3D map, the SLAM process can continue in step 812 until a second new map is created. Then, as shown in step 814, the second new map can be merged with the fused 3D map to create a cooperative 3D map. However, if the fused 3D map includes features of the newly drawn map, method 800 can terminate in terminator 816.

[0125] In some other embodiments, the method further includes sharing a collaborative 3D map with a persistent virtual world system, wherein multiple features of the collaborative 3D map are shared with the persistent virtual world system as part of the virtual object.

[0126] Figure 9 A block diagram of method 900 is depicted, in which image data is simultaneously received from first and second client devices, realizing an updatable collaborative 3D map data fusion platform and its persistent virtual world system.

[0127] Method 900 can begin in step 902 and be executed. Figure 7The actions in steps 702-708 are then performed. Afterwards, in step 904, method 900 can continue to obtain the poses of the first and second client devices. In step 906, the method can continue to simultaneously perform a SLAM process in the first and second coordinate spaces by capturing image data from at least one imaging device of the first and second client devices, wherein the SLAM process is performed by determining poses with first and second plurality of features in the first and second coordinate spaces using the poses of the first and second client devices. Then, in step 908, method 900 can continue to check whether enough features have been extracted so that, in step 910, features can be compared and matched. If the first and second plurality of features match, method 900 continues to merge the first and second feature maps, wherein the merging is performed in alignment with a base map. Subsequently, method 900 continues to compare and merge the combined map with the RVRN to generate a collaborative 3D map.

[0128] If the first and second feature maps do not match, method 900 can be performed in step 914 to compare and merge the first and second feature maps with the RVRN to generate a collaborative 3D map. After step 912 or 914, method 900 can end in terminator 916.

[0129] Figure 10 A block diagram of a method 1000 according to one embodiment is depicted, which illustrates further steps in a method for implementing an updatable collaborative 3D map data fusion platform and its persistent virtual world system.

[0130] Method 1000 may begin at step 1002 by obtaining an identification code from a first or second client device. The personal ID code may be, for example, a QR code, URL, IP address, MAC address, cryptographic hash, universally unique identifier, or organization-unique identifier. The ID code is attached to the 3D coordinates of the total area mapped by each client device. Then, at step 1004, the method may continue by associating the identification code with the first or second map, respectively. Finally, method 1000 may end at step 1006 by storing the identified first or second map in a smart contract implemented in a distributed ledger. In yet another embodiment, the map area contribution identified by the client can be used in a reward system that includes providing rewards associated with the map area contribution of the client device.

[0131] Figure 11 A block diagram according to one embodiment is depicted, illustrating further steps in a method for implementing an updatable collaborative 3D map data fusion platform and its persistent virtual world system.

[0132] Method 1100 may begin in step 1102 by selecting one or more virtual objects stored in the memory of at least one server computer via a first or second client device. Then, in step 1104, method 1100 may end by overlaying the selected virtual objects onto one or more locations using a collaborative 3D map shared with the persistent virtual world system as a reference for the alignment of the virtual objects.

[0133] The document also describes a computer-readable medium having instructions stored thereon, the instructions being configured to cause one or more computers to perform any of the methods described herein. The computer-readable medium may include volatile or non-volatile, removable or non-removable media implemented in any method or technology capable of storing information, such as computer-readable instructions, data structures, program modules, or other data. Typically, the functionality of the computing device described herein can be implemented in computational logic implemented in hardware or software instructions that can be written in a programming language, such as C, C++, COBOL, or JAVA. TM , PHP, Perl, Python, Ruby, HTML, CSS, JavaScript, VBScript, ASPX, Microsoft.NET TM Languages ​​(e.g., C#) and / or similar languages. The computational logic can be compiled into an executable program or written in an interpreted programming language. Typically, the functionality described herein can be implemented as a logic module, which can be replicated to provide greater processing power, merged with other modules, or divided into submodules. The computational logic can be stored in any type of computer-readable medium (e.g., non-transitory medium, such as memory or storage medium) or computer storage device, and can be stored on and executed by one or more general-purpose or special-purpose processors, thereby creating a special-purpose computing device configured to provide the functionality described herein.

[0134] While certain embodiments have been described and illustrated in the accompanying drawings, it should be understood that these embodiments are merely illustrative and not limiting of the invention, and that the invention is not limited to the specific constructions and arrangements shown and described, as various other modifications will be apparent to those skilled in the art. Therefore, this description is to be considered illustrative rather than restrictive.

Claims

1. A method for implementing a collaborative 3D map data fusion platform, characterized in that, include: A 3D map data fusion platform is provided in the memory of one or more server computers, including at least one processor; The 3D map data fusion platform provides a basic satellite map of the world location, which is obtained by using satellite imagery mapping technology; The 3D map data fusion platform references the basic satellite map to publish a detailed, real-time virtual copy network of real-world locations. Obtain the attitude of the first client device; Perform synchronous localization and mapping SLAM processing in a first coordinate space, wherein the SLAM processing uses image data from at least one imaging device of the first client device, the pose of the first client device, and data from the base satellite map to determine the pose of a first plurality of features in the first coordinate space; Create a first new map, which includes the three-dimensional coordinates of the first plurality of features within the first coordinate space; The 3D map data fusion platform merges the first new map, which includes the three-dimensional coordinates of the first plurality of features within the first coordinate space, with the published real-time virtual copy network to create a fused 3D map. Obtain the attitude of the second client device; Initiate SLAM processing in a second coordinate space, wherein the SLAM processing in the second coordinate space uses image data from at least one imaging device of the second client device, the pose of the second client device, and the data from the fused 3D map to determine the pose of a second plurality of features within the second coordinate space; Create a second new map that includes the second plurality of features; and The 3D map data fusion platform merges the second new map with the fused 3D map to create a collaborative 3D map; and The first client device and the second client device include SLAM scanning and navigation modules to preprocess the image data.

2. The method according to claim 1, characterized in that, Creating the first new map includes comparing the first plurality of features with the published real-time virtual copy network by the 3D map data fusion platform through one or more feature matching algorithms.

3. The method according to claim 2, characterized in that, Once the threshold number of features in the first coordinate space has been reached, a comparison step is performed.

4. The method according to claim 1, characterized in that, It also includes sharing the collaborative 3D map with a persistent virtual world system that includes virtual copies, purely virtual objects, applications, or combinations thereof, wherein objects drawn on the new map are added to the persistent virtual world system.

5. The method according to claim 1, characterized in that, Also includes: Select one or more virtual objects through the first or second client device; and The first or second client device overlays the virtual object on top of one or more locations by using the collaborative 3D map shared with the persistent virtual world system as a reference for aligning the selected virtual object.

6. The method according to claim 1, characterized in that, Also includes: Obtain the identification code from the first or second client device; Associate the identification code with the first new map or the second new map; and The map associated with the identifier is stored in a smart contract implemented in a distributed ledger.

7. The method according to claim 6, characterized in that, The stored map is used in a reward system that includes providing rewards related to map area contributions from the first or second client device.

8. The method according to claim 1, characterized in that, It also includes adding each user's rights information to the image data, wherein only authorized users have the right to view the image data for the location.

9. The method according to claim 1, characterized in that, The SLAM processing is performed in part by one or more processors of the first client device and the one or more server computers.

10. A system for implementing a collaborative 3D map data fusion platform, characterized in that, include: At least one server computer, including a memory and at least one processor, wherein the memory stores a 3D map data fusion platform, and the at least one server computer is configured to: The 3D map data fusion platform provides a basic satellite map of the world location, which is obtained by using satellite imagery mapping technology; The 3D map data fusion platform references the basic satellite map to publish a detailed, real-time virtual copy network of real-world locations. Obtain the attitude of the first client device; Perform synchronous localization and mapping SLAM processing in a first coordinate space, wherein the SLAM processing uses image data from at least one imaging device of the first client device, the pose of the first client device, and data from the base satellite map to determine the pose of a first plurality of features in the first coordinate space; Create a first new map, which includes the three-dimensional coordinates of the first plurality of features within the first coordinate space; The 3D map data fusion platform merges the first new map, which includes the three-dimensional coordinates of the first plurality of features within the first coordinate space, with the published real-time virtual copy network to create a fused 3D map. Obtain the attitude of the second client device; Initiate SLAM processing in a second coordinate space, wherein the SLAM processing in the second coordinate space uses image data from at least one imaging device of the second client device, the pose of the second client device, and the data from the fused 3D map to determine the pose of a second plurality of features within the second coordinate space; Create a second new map that includes the second plurality of features; and The 3D map data fusion platform merges the second new map with the fused 3D map to create a collaborative 3D map; and The first client device and the second client device include SLAM scanning and navigation modules to preprocess the image data.

11. The system according to claim 10, characterized in that, Creating the first new map includes comparing the first plurality of features with the published real-time virtual copy network by the 3D map data fusion platform through one or more feature matching algorithms.

12. The system according to claim 11, characterized in that, Once the threshold number of features in the first coordinate space has been reached, a comparison step is performed.

13. The system according to claim 10, characterized in that, It also includes sharing the collaborative 3D map with a persistent virtual world system that includes virtual copies, purely virtual objects, applications, or combinations thereof, wherein objects drawn on the new map are added to the persistent virtual world system.

14. The system according to claim 10, characterized in that, The at least one server computer is also configured to: Obtain the identification code from the first client device; Associate the identifier with the first new map; and The first new map is identified and stored in a smart contract implemented in a distributed ledger, wherein the stored and identified map is used in a reward system that includes providing rewards related to map area contributions by the first client device.

15. The system according to claim 10, characterized in that, The first client device is configured to select one or more virtual objects stored in the memory of the at least one server computer, and to overlay the selected one or more virtual objects on top of one or more locations using the fused 3D map as a reference.

16. A non-transitory computer-readable medium having computer-executable instructions stored thereon, characterized in that, The computer-executable instructions are configured to cause one or more processors to perform the following steps: Provides a 3D map data fusion platform; The 3D map data fusion platform provides a basic satellite map of the world location, which is obtained by using satellite imagery mapping technology; The 3D map data fusion platform references the basic satellite map to publish a detailed, real-time virtual copy network of real-world locations. Obtain the attitude of the first client device; Perform synchronous localization and mapping SLAM processing in a first coordinate space, wherein the SLAM processing uses image data from at least one imaging device of the first client device, the pose of the first client device, and data from the base satellite map to determine the pose of a first plurality of features in the first coordinate space; Create a first new map, which includes the three-dimensional coordinates of the first plurality of features within the first coordinate space; The 3D map data fusion platform merges the first new map, which includes the three-dimensional coordinates of the first plurality of features within the first coordinate space, with the published real-time virtual copy network to create a fused 3D map. Obtain the attitude of the second client device; Initiate SLAM processing in a second coordinate space, wherein the SLAM processing in the second coordinate space uses image data from at least one imaging device of the second client device, the pose of the second client device, and the data from the fused 3D map to determine the pose of a second plurality of features within the second coordinate space; Create a second new map that includes the second plurality of features; and The 3D map data fusion platform merges the second new map with the fused 3D map to create a collaborative 3D map; and The first client device and the second client device include SLAM scanning and navigation modules to preprocess the image data.

17. The non-transitory computer-readable medium according to claim 16, characterized in that, Creating the first new map includes comparing the first plurality of features with the published real-time virtual copy network by the 3D map data fusion platform through one or more feature matching algorithms.

18. The non-transitory computer-readable medium according to claim 17, characterized in that, Once the threshold number of features in the first coordinate space has been reached, a comparison step is performed.

Images