Achieving virtual intelligence and optimization through multi-source real-time and context-aware real-world data

Abstract

A system and method for providing multi-source, real-time, and context-aware real-world data for machine learning algorithms in artificial intelligence applications, including providing a server and connecting the server and multiple interconnected elements therebetween via a network. Each connected element includes one or more sensing mechanisms. The server includes a memory and a processor. The memory stores a persistent virtual world system, including virtual copies of real-world entities created and edited via a virtual copy editor and updated with multi-source sensing data captured by the sensing mechanisms. Each virtual copy includes data and instructions of the multi-source sensing data. The processor can be set to prepare the data to generate a machine learning data set, then execute a machine learning algorithm on the data set, while generating a trained machine learning model for overall inference of new data and optimizing a system with new world entities.

Description

BACKGROUND OF THE DISCLOSURE

[0001] Aspects of the present disclosure relate generally to computer architectures, and more particularly, to systems and methods for providing machine learning algorithms with multi-source, real-time, and context-aware real-world data for artificial intelligence applications. Such systems and methods may serve as solutions for managing and optimizing real-world elements by processing their corresponding virtual-world counterparts.

[0002] Advances in artificial intelligence or virtual intelligence have been enabled by the application of computing power in machine learning and the increasing availability of large datasets. In machine learning, a programmer provides a computer with a set of sample data and a desired result, and the computer then derives its own algorithms based on this data, which can be applied to any future data. Recent deep learning techniques or machine learning simulations in large data collection have mimicked the complexity of the human brain's structural algorithms and functions in so-called "artificial neural networks."

[0003] Multiple practical types of machine learning include supervised learning, which involves using an input to map an output based on input / output sample pairs. The input-output sample pairs include labeled training data, which typically requires a high degree of human involvement in the process of labeling such data. For example, in the case of training a driverless vehicle, a large dataset representing millions of driving hours of video over several hours needs to be labeled by humans. This increases the cost and time required in this process.

[0004] Other drawbacks of current machine learning and inference techniques include using fragmented datasets from only a few sources, which can result in restrictive decisions that ignore background information, without which overall machine learning and inference would be limited or impossible. Returning to the driverless example, a vehicle can capture data specifically for learning driverless and applying it in the inference process. However, on the other hand, this data is considered limited as it may not take into account factors such as citizen behavior, vehicle priority, date of the year, time of day, etc. Therefore, the decisions made are limited to what the vehicle can see through its sensing mechanisms, which requires particularly fast response and processing speeds; on the other hand, the captured data is only applied in the training of the vehicle, ignoring its potential use in other aspects.

[0005] In addition, generally, virtual models of buildings, manufacturing plants, communities, cities, etc. mainly include shape data for simulating the appearance of these entities, lacking other relevant data that prevents managing and adjusting machine learning and inference for their corresponding real-world entities.

[0006] Accordingly, there is a need to improve the fields of machine learning and inference that can acquire and process large datasets and optimize real-world entities without excessive human intervention and in the context of overall decision-making. Summary of the Invention

[0007] This summary introduces a series of concepts in a simplified form that will be further described in the following detailed description. The purpose of this summary is not to identify the key features of the claimed subject matter nor to assist in determining the scope of the claimed subject matter.

[0008] The present disclosure solves the problems set forth above by providing a system and method for a machine learning algorithm with multi-source, real-time, and context-aware real-world data for artificial intelligence applications. This system and method can serve as a solution for managing and optimizing real-world elements through their corresponding virtual-world counterparts.

[0009] A system of the present disclosure includes a server computer system and multiple elements interconnected via a network with a server computer system. The server computer system includes one or more server computers that include a memory and a processor, and the interconnected elements include sensing mechanisms configured to capture sensing data or multi-source sensing data from multiple sources. A persistent virtual world system stored in the memory includes virtual copies of real-world entities stored in a database or data structure, and these virtual copies are linked to the real-world entities through the sensing mechanisms connected to the server. In addition to data directly corresponding to real-world elements, the multi-source sensing mechanisms can capture background data from the surrounding environment, which is classified as micro-context and macro-context. By providing the sensing mechanisms that continuously capture data from the real world to the multiple connected elements, multi-source sensing data that is real-time and reflects real-world conditions can update the virtual world and the virtual copies. This data can provide valuable information for training machine learning algorithms, which can derive trained machine learning models to achieve a comprehensive inference for new data and subsequent data analysis, thereby obtaining impact data for optimizing the real world. In additional embodiments, the virtual copies are further updated based on user input, server computer system calculations, or a combination of both.

[0010] According to one embodiment, the memory further includes a replica editor, which may include software and hardware configured to enable a user to model and edit virtual replicas of real-world entities. The virtual replica editor may be, for example, a computer-aided drafting software that can store the required data and instructions, input, and edit the virtual replicas. The virtual replica editor can implement the input of clear data and instructions associated with each digital virtual replica, such as data and instructions describing shape, location, position, and orientation, physical properties, and overall descriptions of each virtual replica and the expected functions and impacts of the system.

[0011] Modeling techniques that use explicit data and instructions to transform real-world entities into virtual replicas and provide them in a virtual world system can be based on existing computer-aided design models of real-world entities. In other embodiments, before integrating real-world entities into an ongoing virtual world system, radar imaging, such as synthetic aperture radar, real aperture radar, AVTIS radar, light detection and ranging (LIDAR), inverse aperture radar, monopulse radar, and other types of imaging techniques can be used to reflect and model real-world entities.

[0012] In addition to the shape and other characteristics of real-world elements, the explicit data and instructions input through the replica editor can include descriptive data and instructions that detail the expected functions and behaviors of real-world elements, including, for example, expected electricity and water consumption. This information can be used to obtain an expected impact (such as a carbon footprint), and once the real-world entity is in operation, this information can be used to compare with the measured impact, thereby enabling optimization of the functions of real-world elements and systems, including real-world entities, through the systems and methods of the present disclosure.

[0013] According to one embodiment, the multi-source sensing data may also include scenario data, such as micro-scenarios and macro-scenarios, which can be transmitted to an ongoing virtual world system to become a digital micro-scenario and a digital macro-scenario, respectively. The digital scenario data can be updated in real time based on the multi-source sensing data obtained by the sensing mechanisms of interconnected elements.

[0014] The processor is configured to process data and instructions from memory, including preparing data using multi-source sensing data and implicit data for generating sensing and implicit data sets; simulating the sensing and implicit data sets to derive additional machine learning-based input data sets; performing machine learning-based training by executing machine learning algorithms on the machine learning-based input data sets to derive a trained machine learning-based model for inferring new data; performing machine inference on new data using the trained data set to obtain implicit data; performing impact analysis on the implicit data to obtain impactful data; and simultaneously optimizing the real world. In the present disclosure, the term "implicit" data refers to data measured after the operation of the system, which reflects the actual behavior of the system rather than the expected behavior (e.g., precise data).

[0015] Data preparation performed by the processor is the process of converting raw data into a machine-usable data set that can be used for machine learning data sets. Data preparation may include known techniques in the art, such as data preprocessing for converting raw data into a clean data set, and using techniques such as cleaning data, integrating data, transforming data, and reducing data; and using data contention in converting raw data into a format suitable for the machine learning process, while techniques such as extracting data, classifying data, decomposing data into a suitable structured format, and storing data may be used. In some embodiments, data preparation may additionally include identity descriptors included in the persistent virtual world system. The identity descriptors may include descriptive information on multiple real-world entities. The descriptive information may be input through a copy editor or captured by a sensing mechanism during the derivation and update of the persistent virtual world system. Therefore, this data belongs to data that can be automatically extracted from the system without human intervention. Preparing data is preferably performed through existing automated machine learning in the art to reduce or eliminate human intervention during the processing.

[0016] The processor may perform simulation to obtain additional machine learning input data sets for training a machine learning algorithm. Since the data included in the persistent virtual world system includes rich scenario-based real-time data from multiple sources, the simulation performed can be more accurate than simulations performed using data sets from limited and fragmented sources.

[0017] According to one embodiment, the machine learning algorithms include, but are not limited to, any combination of Naive Bayes classification algorithm, nearest neighbor algorithm, K-means algorithm, support vector algorithm, Apriori algorithm, linear regression algorithm, logistic regression algorithm, neural network algorithm, random forest algorithm, decision tree algorithm, etc. Additionally, the machine learning algorithms can be combined with one or more of supervised learning, unsupervised learning, and reinforcement learning. In some embodiments, the supervised learning algorithms can use a dataset including identifiers included in the persistent virtual world system. These identifiers can be automatically attached by the processor to the multi-source sensing data preparation and can be equivalent to the data labels generally used in supervised machine learning algorithms. Therefore, since the data in the virtual world system is already identified data, the step of labeling generally performed by humans can be replaced by the processor and carried out automatically.

[0018] Training machine learning algorithms with rich scenario real-world sensing data, implicit data, and simulation data can provide a well-trained machine learning algorithm for use in inferring data and managing real-world entity operations. By the trained machine learning algorithm, taking into account the scenario of each real-world element and the time required to generate a visual impact, using the system of the present disclosure to infer new data can derive implicit data reflecting the real behavior of a system. Therefore, once there is enough implicit data, the processor can analyze and evaluate the real impacts caused during operation (such as using the system and method of the present disclosure to manage the operation of a building). While generating a virtual copy, this impact data can be compared with the impact data originally included in the implicit data. Here, the system including the machine learning algorithm can be improved and optimized.

[0019] According to one embodiment, by applying the trained machine learning model, the orientation of organizing or operationalizing machine learning and inference can be achieved. Organizational training and inference can be performed by using data to fill the machine learning algorithms. In addition to implicit data and multi-source sensing data, this data also includes the organizational data required for managing the persistent virtual world system. Using this data to train the algorithms and simultaneously deriving a trained machine learning model that can be used in organizational management. In some embodiments, the organizational learning and inference can be goal-oriented, meaning that the organizational data used to train the machine learning algorithms can be trained considering one or more specific goals. To be consistent with this goal, the actions generated by the machine learning algorithms can form a system. For example, it can include reducing energy consumption where energy is not needed, optimizing traffic conditions, and reducing carbon dioxide emissions from each means of transportation.

[0020] Operationalizing machine learning and inference can be achieved by feeding a machine learning algorithm with a specific subset of data that includes the functions necessary to operate individual real-world entities, using this data to train the machine learning algorithm, and deriving a trained machine learning model that can be used in the operation of a device. In some embodiments, organizational learning and inference can be performed by a server computer system for the purpose of coordinating the operational inference of individual interconnected elements of a system. Operational learning and inference can be performed separately by the interconnected elements, by the server computer system, or by a combination thereof, while the server computer system sends and uses data to coordinate the operation of the individual interconnected elements of the system.

[0021] By providing input data and instructions from multiple data sources (such as multi-source sensing data, explicit data, analog data, implicit data, and impact data) to a central structure (such as a server) and storing and updating this data in a continuous virtual world system replicated from the real world, machine learning algorithms can be comprehensively trained for multiple applications. Thus, the way of making decisions using a trained machine learning model, such as inference of new data, can be performed by using multiple resources, ultimately forming an overall approach that not only considers data directly related to the target entity or target data but also considers the situational data around the target entity.

[0022] The persistent virtual world system includes continuously updated data that can be accessed at any time and is structured data recognized within the system. Multiple entities in the real world are continuously tracked not only by sensing mechanisms installed on multiple interconnected elements but also by using transceiver units installed in interconnected devices. The transceiver sends data to and receives data from the antenna and is affected by multiple existing tracking technologies. Combining the tracking implemented by the transceiver with certain sensing mechanisms, particularly inertial sensing devices that provide accurate direction, speed, acceleration, and other parameters of interconnected elements, enables activation of the tracking devices included in the system. In one embodiment, data on the position and orientation of each entity has been input into the system, and the system also recognizes this data and other descriptive identification data, eliminating the need to manually label data to provide labeled data for supervised training algorithms, thus enabling a faster and more cost-effective machine learning algorithm training method. Additionally, as data, particularly data on position and orientation, is provided in the system in real time, since the device does not need to first sense and track the positions of these other entities for subsequent data inference, it can use trained machine learning models to infer new data. Further, since the persistent virtual world system captures service context and material information from users, such as wall widths, window positions, building positions, and other infrastructure details of buildings, the wireless signals of the radio access network from the base station take this information into account and can make relevant adjustments to optimize QoS.

[0023] According to one embodiment, the antenna can be configured to transmit and receive radio waves to enable mobile communication with system elements such as interconnected elements and servers. The antenna can be connected to the computing center or data center where the server computer is located wirelessly or wiredly. In another embodiment, the antenna is provided in the computing center and / or in the area served by the computing center. In some embodiments, to provide connectivity to computing devices located outdoors, the antenna can include a millimeter-wave-based antenna system or a combination of a millimeter-wave and sub-6GHz antenna system, such as through fifth-generation wireless system communication (5G). In another embodiment, the antenna can include other types of antennas, such as a 4G antenna that can serve as a support antenna for a millimeter-wave / sub-GHz antenna system combination. In an embodiment where the antenna is used to provide connectivity to indoor interconnected elements, the antenna can use a wireless local area network, preferably but not limited to providing data at 16GHz.

[0024] According to one embodiment, the sensing mechanism installed on interconnected elements includes an inertial tracking sensing mechanism and an integration with a transceiver. The inertial tracking sensing mechanism can use devices such as accelerometers and gyroscopes, which can be integrated into an inertial measurement unit. A transceiver can be used to send wireless communication signals to an antenna and receive wireless communication signals from the antenna. Preferably, the transceiver is a millimeter-wave transceiver. In an embodiment using a millimeter-wave antenna, the millimeter-wave transceiver is arranged to receive millimeter-wave signals from the antenna while sending data back to the antenna. The inertial sensor and position tracking provided by the millimeter-wave transceiver, along with the precise tracking, low latency, and high QoS functions provided by the millimeter-wave-based antenna, can achieve sub-centimeter or sub-millimeter position and orientation tracking, thereby improving accuracy while tracking the real-time position and orientation of interconnected elements. In some embodiments, several techniques known in the art can be used to implement tracking, such as time of arrival, angle of arrival, or other tracking techniques known in the art (such as visual imaging, radar technology, etc.). In other embodiments, the sensing mechanism and the transceiver can be coupled in a single tracking module device.

[0025] Providing an accurate tracking of interconnected elements is useful for presenting a reliable state of entities within the scope of a persistent virtual world system, especially information related to the position and orientation associated with multiple applications. Additionally, achieving an accurate real-time tracking of interconnected elements can reduce the physical sensing requirements for other interconnected elements in the system before data inference and corresponding decisions are made. However, in cases where a server computer system needs to act based on the presence of non-connected elements or other entities that have not yet been stored in the persistent virtual world system, such as people, animals, trees, or other elements, it is still necessary to use certain sensing mechanisms, such as cameras.

[0026] According to one embodiment, through a distributed ledger-based communication pipeline connected to a network, the transceiver enables communication between computing devices, thereby enabling communication between connected elements. Situations that require direct communication between interconnected elements and a detour through a server include emergency situations where decisions need to be made within a very short period of time.

[0027] According to one embodiment, a computer-implemented method enables providing multi-source, real-time, and context-aware real-world data for use in machine learning algorithms, which can be used as a solution for managing and optimizing real-world entities. The method begins by providing a server computer system including one or more server computers, each server computer including a processor and a memory, where the processor is configured to execute instructions and data stored in the memory, and the memory includes a persistent virtual world system storing virtual copies of real-world entities. According to one embodiment, the virtual copies are derived by a copy editor that can input explicit data for each virtual copy.

[0028] The method continues by providing a plurality of computing devices connected to the server computer system via a network, each interconnected element including one or more sensing mechanisms. The sensing mechanisms then capture multi-source sensing data from real-world entities to enrich and update the virtual copies included in the persistent virtual world system and then transmit the data to the server computer system. The multi-source sensing data includes data captured by real-world elements and background data including micro-context data and macro-context data. In some embodiments, the interconnected elements may be additionally connected via a network. In other embodiments, the interconnected elements may be interconnected with each other or with the server via a distributed ledger.

[0029] The method then continues with the server computer system preparing a processor including explicit data and multi-source sensing data. The resulting data set is then used during a simulation providing additional data sets. In one embodiment, the method continues by training a machine learning algorithm using a machine learning input data set, where the input data set includes a simulation data set, an explicit data set, and a multi-source sensing data set, to derive a trained machine learning module. The method then continues by using the trained machine learning model to infer new data and derive implicit data. The method then checks if there is sufficient implicit data. In the negative case, where there is not enough implicit data, the method returns to training the machine learning algorithm, taking into account feedback information from the implicit data. In the case where there is sufficient data available, the method continues by analyzing the data to obtain ways to influence the data. The influence data is then used to optimize real-world entities, with the goal of reducing negative impacts.

[0030] The above summary does not include all aspects of the present disclosure. All systems and methods included in the present disclosure can be implemented according to all suitable combinations of the aspects summarized above and the aspects disclosed in the following detailed description, particularly the claims pointed out in the present application. This combination has particular advantages that are not mentioned in the above summary. Other features and advantages will be presented in the drawings and the following detailed description. Brief Description of the Drawings

[0031] The following detailed description in conjunction with the drawings can better understand the aspects mentioned above and more subsequent advantages. It includes:

[0032] Attached Figure 1 A schematic diagram of a system implementing a machine learning algorithm that provides real-world data with multi-source, real-time, and context awareness is described.

[0033] According to an embodiment of the present disclosure, attached Figure 2 A schematic diagram of a system is described, which details a server connected to interconnected elements;

[0034] According to an embodiment of the present disclosure, attached Figure 3 A schematic diagram of a system is described, which details a representative of interconnected elements; and

[0035] Attached Figure 4 A schematic diagram of a method is described, which can provide a machine learning algorithm for real-world data with multi-source, real-time, and context awareness expressed by the interaction between a processor and a memory in a server.

[0036] Attached Figure 5 A method is described, which can provide a machine learning algorithm for real-world data with multi-source, real-time, and context awareness. Detailed Description of the Embodiments

[0037] In the following description, reference is made to the drawings, and various embodiments are shown by way of example. At the same time, multiple embodiments will be described by referring to several examples. Without departing from the scope of the claimed subject matter, the embodiments include changes in design and structure.

[0038] Attached Figure 1Disclosed is a schematic diagram of a system 100 that can be configured to provide machine learning algorithms with multi-source, real-time, and context-aware real-world data for artificial intelligence applications, and to solve the problems of managing and optimizing real-world entities by processing their corresponding virtual-world counterparts. System 100 includes a server 102 connected via a network 104 to a plurality of interconnected elements 106 for capturing multi-source sensor data 108 via a plurality of sensing mechanisms (not shown). A plurality of sensing mechanisms installed on the interconnected elements 106 capture the context from themselves and from each connected element 106, including any real-world elements. Accordingly, each real-world element may or may not be an interconnected element 106. For example, a building can be both an interconnected element 106 and a real-world element, while a tree may only represent a real-world element but not an interconnected element 106.

[0039] The plurality of interconnected elements 106 can include one or more mobile phones, laptops, wearable computers, personal computers, mobile electronic game consoles, smart contact lenses, head-mounted display devices, see-through devices, surveillance cameras, vehicles, traffic lights, buildings and other structures, streets, train tracks, household appliances, or other devices connected via a network 104, or any appliance including such devices. According to one embodiment, the plurality of sensing mechanisms installed on the interconnected elements 106 include one or more temperature sensors, proximity sensors, inertial sensors, infrared sensors, pollution sensors (such as gas sensors), pressure sensors, light sensors, ultrasonic sensors, smoke sensors, touch sensors, chromaticity sensors, humidity sensors, water sensors, or electrical sensors, or a combination thereof.

[0040] Although the appended Figure 1 shows a server 102, for easier understanding, one or more servers 102 at one or more locations can be incorporated into a server computer system to serve a complex system including a plurality of connected elements 106 located at different locations. Accordingly, the plurality of interconnected elements 106 can be connected to one or more servers 102 according to the location of the interconnected elements 106 at a particular time. Additionally, any example described herein with reference to a single server 102 can also be implemented with multiple servers. Additionally, as shown in the appended Figure 1 figure, the plurality of interconnected elements 106 can be connected to the server 102 and can also be connected to each other via the network 104.

[0041] Multi-source sensing data 108 includes capturable data of real-world elements, including 3D imaging data, 3D geometric data, 3D entities, 3D sensing data, 3D dynamic objects, video data, audio data, priority data, chemical components, waste generation data, text data, time data, location data, orientation data, speed data, temperature data, humidity data, pollution data, light data, volume data, flow data, color data, energy consumption data, bandwidth data, mass data, or other data captured by sensing mechanisms. Multiple sensing mechanisms capture scenario data from the nearby environment in addition to data directly corresponding to one or more real-world elements, which can be classified as micro-scenarios 110 and macro-scenarios 112.

[0042] The term "real-world element" as used in this disclosure refers to an element found in the real world that can be sensed by a sensing mechanism and is prone to taking actions through machine learning algorithms. Real-world elements can be dynamic or static entities in the real world, including humans, vehicles, buildings, objects, entertainment areas, natural formations, streets, and any other elements that can be found in the real world. Real-world elements captured by the sensing mechanisms of interconnected elements 106 can be extracted from 3D imaging data, 3D geometric data, video data, audio data, temperature data, mass data, radiation data, tactile data, motion data, or other capturable data that can be obtained by sensing mechanisms. Therefore, a real-world element itself can include a sensing mechanism and the multi-source sensing data 108 captured by itself. Thus, for the perception of other interconnected elements, other interconnected elements can also be classified as real-world elements.

[0043] The term "context" or "context data" as used in this disclosure refers to data associated with the direct or indirect environment of a specific real-world element. This data can be classified into "micro-context" and "macro-context".

[0044] The term "micro-context" refers to the context directly surrounding a real-world element, such as a person, an object, or conditions that have a direct impact on the real-world element. Micro-context 110 can include data directly surrounding and affecting a target real-world element, such as 3D imaging data, 3D geometric data, 3D entities, 3D sensing data, 3D dynamic objects, video data, audio data, text data, time data, metadata, priority data, security data, location data, light data, temperature data, and quality of service. In one embodiment, micro-context 110 also includes service context, which here refers to the actual applications used by the user or nearby users. Since an application consumes bandwidth, the service context can provide a valuable information system that needs to evaluate the network signals provided to interconnected elements 106.

[0045] The term "macro background" refers to the indirect or more distant background around real-world elements. The macro background can be read by the server 102 from multiple micro backgrounds 110 to generate more comprehensive system information, such as the current efficiency of a factory, air quality, climate change level, company efficiency, city efficiency, national efficiency, etc. The macro background can be considered and calculated at different levels according to specific machine learning functions and objectives, including the local level (such as an office or manufacturing plant), community level, city level, national level, or even planetary level. Therefore, different types of macro backgrounds 112 can be derived from the same real-world element data and micro background data according to specific machine learning functions and objectives.

[0046] In some embodiments, the network 104 can be a cellular network and can use multiple technologies including Enhanced Data Rates for GSM Evolution (EDGE), General Packet Radio Service (GPRS), Global System for Mobile Communications (GSM), Internet Protocol Multimedia Subsystem (IMS), Universal Mobile Telecommunications System (UMTS), etc., as well as any suitable wireless media, such as Worldwide Interoperability for Microwave Access (WiMAX), Long-Term Evolution (LTE) network, Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Wireless Fidelity (WiFi), satellite, Mobile Ad-hoc Network (MANET), etc.

[0047] According to one embodiment, an antenna (not shown) is provided for transmitting and receiving radio waves that can enable mobile communication with elements of the system 100 (such as interconnected elements 106 and server 102). The antenna can be connected to the computing center or the data center where the server is located by wired or wireless means. In other embodiments, the antenna is provided within the scope of the computing center and / or the area served by the computing center. In some embodiments, to provide connectivity to outdoor computing devices, the antenna can include a millimeter-wave-based antenna system or a combination of a millimeter-wave-based antenna and a sub-6 GHz antenna system, such as through 5G (5th Generation Wireless System). In other embodiments, the antenna can include other types of antennas, such as 4G antennas, which can serve as support antennas for the millimeter-wave / sub-GHz antenna system. In embodiments where the antenna is used to provide connectivity to indoor computing devices, the antenna can use, but is not limited to, a wireless local area network that provides 16 GHz data.

[0048] The sensing mechanism can be part of the tracking module (not shown), which can include a transceiver. The sensing mechanism used in the tracking module can include inertial sensors that provide accurate data regarding direction, speed, acceleration, and other parameters of the interconnected element 106. The transceiver of the tracking module sends data to the antenna and receives data from the antenna, while being limited by various existing tracking technologies. In one embodiment, precise tracking of the interconnected element 106 included in the system can be achieved in combination with the tracking performed by the transceiver having inertial sensors.

[0049] According to one embodiment of the present disclosure, attached Figure 2 A schematic diagram of a system 200 is described, which details a server 102 communicatively connected to the interconnected element 106 through a network 104. Attached Figure 2 Some of the elements may be similar to the elements of attached Figure 1 and thus similar or identical reference numerals are used to describe those elements.

[0050] The system 200 of the present disclosure includes a server 102 and a plurality of interconnected elements 106 connected to the server 102 through a network 104. The server 102 includes a memory 202 and a processor 204. Although for simplicity of description, the server 102 is described as having a single memory 202 and processor 204, it is understood that the server can include multiple processors 204 and multiple memories 202. Thus, any example described herein involving a single processor 204 or memory 202 can also be implemented with multiple processors 204 and / or multiple memories 202.

[0051] The memory 202 stores a virtual world system 202 in a database or data structure. The memory 202 also includes multiple virtual copies 208, such as virtual copies 208A, B, and C corresponding to respective display world elements. The virtual copies 208 are communicatively connected to real-world elements through a sensing mechanism connected to the server 102 via the network 104. Certain virtual copies 208, such as those corresponding to elements that do not include a sensing mechanism (such as natural formations, old vehicles, old buildings, new elements not yet connected to the server 102, etc.), i.e., virtual copies 208 that do not include any sensing mechanism (such as natural formations, old vehicles, old buildings, new elements not yet connected to the server 102, etc.), can only appear graphically in the persistent virtual world system 206 and can only be visually updated through optical sensors attached to other interconnected elements 106, such as cameras.

[0052] In the present disclosure, the term "persistent" may be used to define a state of a system that can continue to exist without continuously performing processing and network connectivity. For example, the term "persistent" is used to define a virtual world system, including the virtual world system, all virtual copies, complete virtual objects, and digital reality applications within it. After the process of creating virtual copies stops for complete virtual objects and digital reality applications, they continue to exist after an independent user connects to the virtual world system. Thus, the virtual world system is stored in a non-volatile storage location, such as on a server. In this way, when set to accomplish a specific goal, virtual copies, complete virtual objects, and digital reality applications can interact and collaborate even when the user is not connected to the server.

[0053] According to one embodiment, the memory 202 further includes a copy editor 210 that includes software and hardware that can be set to enable a user to model and edit a virtual copy 208 of a real-world entity. The copy editor 210 can be a computer-aided design software that can store the data and instructions required to input and edit the virtual copy 208. The copy editor 210 can implement the input of explicit data and instructions 212 related to the digital copy. In one embodiment, the explicit data and instructions describe the shape, position, location and orientation, physical properties, and the expected functions and impacts of each virtual copy 208 and model the entire persistent virtual world system 206 and for it.

[0054] "Explicit data" here refers to data that cannot be obtained by sensing mechanisms but requires digital input through the copy editor 210, such as priority data, building materials, wall thickness, electrical installations and circuits, water pipes, fire extinguishers, emergency exits, window positions, machine performance parameters, machine sensors, and valve positions, etc. The "priority data" used here refers to a hierarchy of real-world entities. For example, certain vehicles (such as ambulances) or people (such as presidents, government officials, police, etc.) may have a higher priority, and their inferences based on the data will affect the decisions made. The "instructions" used here refer to code (such as binary code) that can be executed by the processor 204. In the context of the virtual copy 208, the instructions represent the behaviors of real-world elements.

[0055] For example, a virtual copy 208 of an elevator may include data and instructions representing the geometry, materials, physics, mechanics, and functions of the elevator. For example, the function of moving from one floor to another can be updated in real time in the persistent virtual world system 206 while the elevator moves in real life. Similarly, the elevator can be indirectly manipulated in real life by operating the virtual copy 208.

[0056] Modeling techniques that convert real-world entities into virtual copies 208 with clear data and instructions and make them available for use in a persistent virtual world system 206, which are based on off-the-shelf computer-aided design models of real-world entities. For example, a machine owner can provide an administrator of a persistent virtual world system 206 or can input an off-the-shelf digital computer-aided design model of their machine themselves. Similarly, a property owner can provide a building information model (BIM) with building details stored in a persistent virtual world system 206, which may include information that is not visible or not easily obtained through a sensing system. In these embodiments, the owners of these real-world entities may be responsible for adding virtual copies in the persistent virtual world system 206, which can be achieved through an incentive mechanism or through legal requirements. In some embodiments, an administrator of the persistent virtual world system 206, or even a government official, can cooperate with the owner of a real-world entity to input the real-world entity into the persistent virtual world system 206, while enabling a faster and more comprehensive creation of the persistent virtual world system 206.

[0057] In other embodiments, before aggregating real-world entities in a virtual world system 206, radar imaging techniques such as synthetic aperture radar, real aperture radar, light detection and ranging (LIDAR), inverse aperture radar, monopulse radar, etc., and other types of imaging techniques can be used to map and model real-world entities. These more technical solutions are particularly used in cases where an initial model of a structure is not available, or where there is a lack of information, or where additional information not provided by a computer-aided design model needs to be added to a persistent virtual world entity.

[0058] The clear data and instructions 212 input through a copy editor 210 can include, in addition to the shape and other characteristics of real-world elements, data and instructions that detail the expected functions of real-world elements, such as including expected electricity and water consumption. Using this information (such as a carbon footprint) to obtain an expected impact can also be used to compare with a measured impact during the operation of a real-world entity and can be used later, through the systems and methods of the present disclosure, to optimize the functions of real-world elements.

[0059] For example, the clear data and instructions 212 for a building can include the shape and characteristics of the building (such as 3D shape, wall thickness, fire alarm location, materials used for each part, window location, wire location, and water pipes, etc.), and details on the amount of water, electricity, gas, and bandwidth that the building is expected to consume, the number of people allowed in the building, the daily footfall, etc. This detailed clear data and instructions can be used to calculate the estimated efficiency and impact of the building. For example, the impact can be represented by the carbon footprint of the building, which refers to the amount of carbon dioxide released into the air due to activities occurring in a particular building. Once the building is in operation and managed by the systems and methods of the present disclosure, the actual carbon footprint can be obtained, and the predicted carbon footprint can be used as a basis for comparing the efficiency of the building, which can then be optimized through machine learning training.

[0060] In one embodiment, independent of the modeling technique used to create the virtual replicas 208, the information for each virtual replica 208 provides sufficient detail about each corresponding real-world element such that highly accurate virtual replicas 208 of each real-world element are available. The virtual replicas 208 can be enriched and updated through multi-source sensing data 108. Thus, the virtual replicas 208 can include data 214 and instructions 216 that are used to describe the true appearance and behavior of each real-world element, except for virtual replicas 208 of non-connected elements, which may only provide their corresponding true appearance and no data and instructions regarding their functionality.

[0061] In the present disclosure, the term "enrich" is used to describe the act of providing additional characteristics to a virtual replica based on multi-source sensing data. For example, enriching a virtual replica can include providing real-world data captured from sensing mechanisms, where the additional real-world data includes video data, temperature data, real-time energy consumption data, real-time water consumption data, etc.

[0062] As described in the append Figure 1 As described, the multi-source sensing data 108 can also include background data, such as micro-backgrounds 110 including micro-backgrounds A, B, and C and a macro-background 112. This same data is then transmitted to the persistent virtual world system 206, making it a virtual micro-background 218 that includes digitized micro-backgrounds A, B, and C, and a virtual macro-background 220 that is updated in real time based on the multi-source sensing data 108 obtained by the connected elements 106. The virtual micro-background 218 and the virtual macro-background 220 also include data 214 and instructions 216 for describing the corresponding real-world appearance and behavior.

[0063] According to one embodiment of the present disclosure, the appendFigure 3 Describes a schematic diagram of a system, which details an interconnected element. Attached Figure 3 Certain elements of Figure 1-2 may be similar to the elements of

[0064] An interconnected element 106 may include, for example, an input / output module 302, a power supply 304, a memory 306, a sensing mechanism that constitutes a tracking module 310, and transceiver 308, as well as operational elements such as a network interface 312, all of which are operatively connected to a processor 314.

[0065] The input / output module 302 is used as computing hardware and software and is configured to interact with a user and provide user input data to one or more other system elements. For example, the input / output module 302 may be configured to interact with a user, derive user input data from the interaction, and then provide the user input data to the processor 314 before being transmitted over a network to other processing systems, such as a server. In another embodiment, the input / output module 302 is used as an external computing pointing device (such as a touch screen, mouse, 3D control, joystick, gamepad, etc.) and / or a text input device (such as a keyboard, dictation tool, etc.) and is configured to interact with other connected elements 106. However, in other embodiments, the input / output module 302 may provide additional, fewer, or different functions than those described above.

[0066] The power supply 304 is used as computing hardware and software and is configured to supply power to the connected element 106. In one embodiment, the power supply 304 may be a battery. The power supply 304 may be housed within the device or may be removable from the device and may be rechargeable or non-rechargeable. In one embodiment, the device may be charged by replacing one power supply 304 with another. In other embodiments, the power supply 304 may be charged by attaching to a charging source wire such as USB, FireWire, Ethernet, Thunderbolt, or a headphone cable. However, in another embodiment, the power supply 304 may be charged by inductive charging, where an electromagnetic field is used to transfer energy from an inductive charger to a power supply 304 when the inductive charger and the power supply are in close proximity and do not require connection by a wire. In another embodiment, a docking station may be used for charging.

[0067] The memory 306 is used as computing hardware and software to adapt to storing application instructions 216, while storing multi-source sensing data 108 captured by multiple sensing mechanisms. The memory 306 can be any medium for storing information that can be accessed by the processor 314, including a computer-readable medium, or other media that can read data via an electronic device, such as a hard disk, memory card, flash drive, ROM, RAM, DVD, or other optical discs, and other writable and read-only memories. The memory 306 can include temporary storage and persistent storage.

[0068] The sensing mechanisms can be used as computing hardware and software to adapt to obtaining various multi-source sensing data 108 from the real world, and determining / tracking the position and orientation of the connected element 106, and one or more real-world elements that may be connected to the connected element 106. The sensing mechanisms can include, but are not limited to, one or more temperature sensors, proximity sensors, inertial sensors, infrared sensors, pollution sensors (such as gas sensors), pressure sensors, light sensors, ultrasonic sensors, smoke sensors, touch sensors, chromaticity sensors, humidity sensors, water sensors, electrical sensors, or combinations thereof. In particular, the sensing mechanisms include one or more inertial measurement units, accelerometers, and gyroscopes. The inertial measurement unit is configured to measure and report the speed, acceleration, angular momentum, translational speed, rotational speed, and other telemetry metadata of the connected element 106 by using the combination of an accelerometer and a gyroscope. The accelerometer within and / or separate from the inertial measurement unit can be configured to measure the acceleration of the interactive device, including the acceleration caused by the Earth's gravitational field. In one embodiment, the accelerometer includes a triaxial accelerometer that can measure acceleration in three orthogonal directions.

[0069] The transceiver 308 can be used as computing hardware and software, configured to enable the device to receive radio waves from the antenna and send data to the antenna. In some embodiments, a millimeter-wave transceiver can be used, which can be used to receive millimeter-wave signals from the antenna and send data back to the antenna in the case of interacting with immersive content. The transceiver 308 can be a bidirectional communication transceiver 308.

[0070] According to one embodiment, transceiver 308 may implement direct communication between computing devices through a distributed ledger-based communication pipeline connected to a network. The distributed ledger-based communication pipeline may implement direct communication between connected elements 106 through a decentralized network, allowing information to be stored in a secure and accurate manner using encryption techniques, encryption keys, and encryption signatures. In cases where communication between connected elements 106 is required, bypassing through a server may include emergency situations where decisions need to be made in a short period of time. For example, in an autonomous driving scenario, when two vehicles are about to collide, direct communication between the two vehicles is required to achieve a faster response to prevent the collision. In another embodiment, a distributed ledger may be used between the server and a connected element 106, where the server may obtain the right to verify data before distributing the data to each connected element 106. In a further embodiment, the distributed ledger may use some connected elements 106 close to the server, close to the antenna, or close to both, with the aim of assigning these connected elements 106 as a central structure to verify and distribute data.

[0071] In one embodiment, the tracking module 310 is implemented by integrating the functions of an inertial measurement unit, an accelerometer, and a gyroscope. At the same time, with the position tracking provided by the transceiver 308 and the tracking, low latency, and high QoS functions provided by the millimeter-wave-based antenna, sub-centimeter or sub-millimeter position and orientation tracking can be achieved, which can improve the accuracy when tracking the real-time position and orientation of the connected element 106. In another embodiment, the sensing mechanism and the transceiver 308 may be coupled in a single tracking module device.

[0072] The network interface 312 may serve as a connection for the computing software and hardware to communicate with the network, receiving computer-readable program instructions 216 sent by the server or other connected elements 106 over the network, and forwarding the computer-readable program instructions 216 for storage in the memory 306 and processing by the processor 314.

[0073] The processor 314 can be used as computing hardware and software and is configured to receive and process multi-source sensing data. For example, the processor 314 can be configured to provide imaging requests, receive imaging data, process the imaging data into environmental or other data, process user input data and / or imaging processing to generate user interaction data, perform edge (device-side) machine learning training and inference, provide server requests, receive server responses, and / or provide user interaction data, environmental data, and content object data to one or more other system components. For example, the processor 314 can receive user input data from the input / output model 302 and can correspondingly execute applications stored in the memory 306. In other examples, the processor 314 can receive multi-source sensing data from sensing mechanisms captured from the real world or receive precise position and orientation information of the interconnected element 106 through the tracking module 310 and can prepare some data before sending the data to the server for further processing. By way of example, the processor 314 can implement the steps required in the data preparation process, including analog or digital signal processing algorithms, such as reducing raw data or filtering the multi-source sensing data 406 before sending the data to the server.

[0074] According to one embodiment, attached Figure 4 FIG. shows a schematic diagram of a system 400, showing the interaction between a processor 204 and a memory 202 for providing multi-source, real-time, and context-aware real-world data to machine learning algorithms. Attached Figure 4 Some elements of may be similar to the elements of attached Figure 2 and thus the same or similar reference numerals are used to describe these elements. The dashed lines represent the order of steps that occur between the processor 204 and the memory 202.

[0075] Referring to the system 400 in attached Figure 4 , the memory 202 is configured to store data 214 and instructions 216 and transfer the data 214 and instructions 216 to the processor 204 for execution as needed. In attached Figure 4 , the processor 204 is described as having functional blocks 402, 412, 418, 422, 426, and 430 for representing programs and paths executed by the processor 204. Although the example described in attached Figure 4 refers to a single processor 204, in operation, this functionality can be performed by one or more processors 204 on one or more computers.

[0076] More specifically, the processor 204 uses the initial data 404, which includes multi-source sensing data 406 and clear data 408, to perform data preparation 402, deriving a sensed and precise data set 410. Then, the processor 204 uses the sensed and precise data set 410 to perform simulation 412, aiming to derive a simulated data set 414 that constitutes an additional machine learning input data set 416. The processor 204 executes a machine learning algorithm on the machine learning input data set 416, aiming to perform machine learning training 418, deriving a trained data set 420 that constitutes a machine learning model for inferring new data. The processor 204 uses the trained data set 420 to perform machine inference 422 on new data, aiming to obtain latent data 424. Impact analysis 426 is performed on the latent data 424, aiming to obtain impact data 428; at the same time, real-world optimization 430 that takes the impact data 428 into account is performed. In the present disclosure, the term "latent" data refers to data that is measured once the system operates, as opposed to the clear data 408, which reflects real data and the expected behavior of the real-world system.

[0077] The data preparation 402 performed by the processor 204 on the multi-source sensing data 406 is a process of converting the raw data into a machine-usable data set, which can be used by applications as a machine learning data set. Data preparation 402 includes techniques known in the art, such as data preprocessing for converting raw data into a clean data set, and techniques such as data cleaning, data integration, data transformation, and data reduction can be used; at the same time, data adaptation can be used to convert the raw data into a suitable format during the machine learning process, using techniques such as data extraction, data classification, decomposing data into a suitable structured format, and storing data. In some embodiments, data preparation 402 additionally includes identity metrics included in the persistent virtual world system. The identity metrics include descriptive information about multiple real-world elements. The descriptive information may have been input through a copy editor in the clear data 408 or captured by the sensing mechanism during the process of deriving or updating the persistent virtual world system 206 through the multi-source sensing data 406, so this data can be automatically extracted from the system without any human intervention. For example, when a car owner adds his car to the persistent virtual world system by inputting the computer-aided design model of the car, the persistent virtual world system can automatically label the car with information such as the car brand, operating performance parameters, and gasoline type. During the data preparation process, some of these identity markers can be automatically extracted when needed and then used in the machine learning training and inference processes. Data preparation can preferably be performed by automated machine learning in the art, which can reduce or eliminate human intervention throughout the process.

[0078] According to one embodiment, the processor 204 performs a simulation 412 using the sensed and sanitized dataset 410, with the aim of deriving an additional machine learning input dataset 416, which can be used during machine learning training 418. Since the data included in the persistent virtual world system includes rich real-time background data from multiple sources, the simulation 412 is more accurate compared to simulations using datasets from limited and fragmented sources. The simulation 412 can use a computer model that includes algorithms and equations used to capture the behavior of the persistent virtual world system. Any existing suitable simulation technique can be used. A simulation 412 can generate a simulated dataset 414 that combines the sensed and sanitized dataset 410, using a large dataset to train machine learning algorithms. Simulated data has particular value in cases where it is difficult to obtain the required dataset, such as in hypothetical scenarios or in cases where data is needed for rare events.

[0079] Machine learning algorithms can include a training and an inference phase. The training and inference of a machine learning algorithm typically involve so-called "tensor operations", or computer operations on multi-dimensional tensors. A multi-dimensional tensor refers to an array of multi-dimensional real numbers. Most of the multi-dimensional tensors involved in a machine learning algorithm belong to a category called "tensor contraction", where two tensors are used as inputs and operations, such as addition and accumulation, are applied to the two tensors to form an output tensor. In some embodiments, the processor can be a central processing unit, a graphics processing unit, or a processor on a chip. In other embodiments, machine learning processing and inference can be performed by combining a central processor, a graphics processor, and a processor on a chip, in any suitable order or scale.

[0080] According to one embodiment, the machine learning algorithm can include any combination of a Naive Bayes classifier algorithm, a nearest neighbor algorithm, an averaging algorithm, a support vector algorithm, a prior algorithm, a linear regression algorithm, a logistic regression algorithm, a neural network algorithm, a random forest algorithm, a decision tree algorithm, etc. Additionally, the machine learning algorithm can combine one or more supervised learning, unsupervised learning, and reinforcement learning. In some embodiments, the supervised learning algorithm can use a dataset that includes identity identifiers included in the persistent virtual world system. These identity identifiers can be automatically appended to the multi-source sensed data 406 during the data preparation 402 performed by the processor 204, and can be equivalent to the data labels commonly used for supervised machine learning algorithms. Thus, the step of labeling, which is usually performed by humans, can be automatically processed by the processor 204, as the data in the persistent virtual world system is already identified data.

[0081] Training a machine learning algorithm using the sensor and clarity dataset 410 and the simulated dataset 414 can provide data for inference during machine inference 422 and autonomously manage the operations of real-world entities through an artificial application. Using the systems and methods of the present disclosure to infer new data with a trained machine learning algorithm can derive implicit data 424 that reflects the real behavior of a persistent virtual world system. Then, after the sufficient time required to generate a visible impact has elapsed and once the implicit data 424 is available, the processor can analyze and evaluate the real impact caused during the operation (e.g., the operation of a building, using the systems and methods of the present disclosure). In the case of generating each virtual copy, this impact data 428 can be compared with the impact information originally included in the input data of the clarity data 408. Using the impact data 428, the processor 204 can achieve real-world optimization 430, which includes using machine learning training and inference focused on increasing real-world benefits, such as mitigating any negative impacts generated by real-world entities. Systems that include the use of machine learning algorithms in the training and inference processes can thus be improved and optimized.

[0082] According to one embodiment, the machine learning and inference through the application of a trained machine learning model are organizationally and operationally directed. Organizational training and inference can be performed by using data to feed a machine learning algorithm, which, in addition to the clarity data 408 and the multi-source sensor data 406, also includes organizational data required for managing a persistent virtual world system. This data is used to train the algorithm and derive a trained machine learning model that can be used in organizational management. In some embodiments, the organized learning and inference can be goal-directed, i.e., the organizational data used to train the machine learning algorithm can be trained while considering one or more specific goals. This specific goal can define a result that can only be achieved in a specific way by all elements in the management system. For example, a specific goal can refer to reducing pollution to a specific percentage in a specific city. The machine learning algorithm is trained such that the machine learning arrangement predicts results consistent with the reduction of pollution in the specific city. The follow-up actions performed by the machine learning algorithm can organize the system to comply with this goal, which can include, for example, consuming less energy when not needed, optimizing traffic conditions, reducing the carbon dioxide emissions of each vehicle, reducing or controlling the number of vehicles circulating at a specific time, replacing or optimizing the materials and processes used in production, turning off unnecessary street lights when no people or vehicles are nearby, etc. Accordingly, since most of the input data used in the machine learning algorithm comes from hypothetical situations where data is difficult to extract, the simulated dataset 414 is of particular importance in goal-directed inference.

[0083] Operational machine learning and inference can be performed by feeding machine learning algorithms with data including a specific subset of functions required to operate individual real-world entities, using this data to train the machine learning algorithms, and a trained machine learning model can be used in device operation. A specific function can be, for example, operating a specific machine, driving a specific autonomous vehicle, operating a specific household appliance, operating a drone, etc. In some embodiments, organizational learning and inference can be performed by a server for the purpose of coordinating the operational inference of individual interconnected elements of a system. Operational learning and inference can be performed individually by one or more interconnected elements, by one or more servers, or a combination of both, and one or more servers can send and use data to coordinate the operation of individual interconnected elements in the system.

[0084] By providing a server with input data (such as a server) and instructions 216 having data from multiple data sources (such as sensed and explicit data set 410, simulated data set 414, implicit data 424, and impact data 428), and storing and updating this data 214 and instructions 216 in a continuous virtual world system replicated according to the real world, machine learning algorithms can be trained holistically for multiple applications. Thus, making decisions using the trained machine learning model can be performed by using multiple sources, forming an overall analysis that considers not only data directly related to the target entity or target data, but also background data related to the vicinity of the target entity. The use of the terms "target" or "target data" in the present disclosure refers to data directly related to real-world elements (such as animals, objects, people, places, etc.) that can be easily identified or acted upon by machine learning algorithms. Generally, any real-world element and its corresponding virtual copy can become a target if the machine learning algorithm has to identify or take another action on that real-world element and its corresponding virtual copy.

[0085] The persistent virtual world system stored in the server's memory 202 includes continuously updated persistent data 214 associated with real-world entities that can be accessed at any time and have been identified within the system. The transceiver sends data to and receives data from the antenna and is affected by the status of several existing tracking technologies. By combining the tracking implemented by the transceiver with certain sensing mechanisms, particularly inertial sensing devices that provide precise direction, speed, acceleration, and other parameters of interconnected elements, precise tracking of the devices included in the system can be achieved. Thus, since the position, orientation data, and other characteristics of real-world elements have been input into the persistent virtual world system, there is no need to label the data, and the labeled data is provided to the supervised training algorithm to implement a faster and more efficient training method for the machine learning algorithm. Additionally, since the data, particularly the position and orientation data, is provided in real time within the virtual world system, the trained machine learning model can be used to infer new data more quickly, while the device does not need to first sense and track the positions of other entities for which the subsequent data is inferred. Moreover, since the persistent virtual world system captures service context and material information from the user, the width of the walls, the position of the windows, the position of the building, and other building infrastructure details, the wireless signals provided by the radio access network at the base station can consider this information and be adjusted accordingly, with the aim of optimizing QoS.

[0086] Providing a precise tracking of interconnected elements is useful for presenting a reliable state of entities within the persistent virtual world system, particularly the position and orientation related to multiple applications. Additionally, achieving a precise, real-time tracking of the connected elements can reduce the physical sensing of other interconnected elements within the system before data inference and corresponding decision-making. For example, for virtual intelligent applications such as autonomous driving, it is important to obtain information about the precise position and orientation of the vehicle or other obstacles within the system, so that a vehicle can identify threats before they become visible as a threat to the vehicle and can respond accordingly. However, in cases where the server needs to act based on the appearance of non-connected elements or other entities that have not yet been stored in the virtual world system, such as people, animals, trees, or other elements, certain sensing mechanisms, such as cameras, are still necessary. Additionally, in some cases, multiple sensing mechanisms are still required to update the state of the virtual world, including multi-source sensing data 406, while providing the multi-source sensing data 406 from the real world required for implementing an overall inference of new data.

[0087] Appendix Figure 5Describes a flowchart of a computer-implemented method 500 that provides multi-source real-time and context-aware real-world data for artificial intelligence applications, which can be used as a solution for managing and optimizing real-world entities. Method 500 can be used in a system, such as the systems described in FIGS. Figure 1 , 2, and 4.

[0088] Method 500 begins at steps 502 and 504 by providing a server that includes a processor and a memory, where the processor is configured to execute instructions stored in the memory and process data stored in the memory, and the memory includes a persistent virtual world system for storing virtual copies of real-world entities. According to one embodiment, the virtual copies are derived by a copy editor that can implement clear data for each virtual copy.

[0089] Method 500 provides, at step 506, a plurality of interconnected elements connected to the server via a network, each interconnected element including one or more sensing mechanisms. Then, at step 508, the sensing mechanisms capture multi-source sensing data from real-world entities, which can enrich and update the virtual copies included in the persistent virtual world system and then transmit the data to the server. The multi-source sensing data includes data captured from real-world elements and context data including micro-context data and macro-context data. In some embodiments, the interconnected elements can be additionally interconnected with each other via the network. In other embodiments, the interconnected elements can be interconnected with each other or with a server via a distributed ledger.

[0090] Then, method 500 proceeds at step 510 by the processor to prepare data including clear data and multi-source sensing data. Then, at step 512, the resulting data set is used in a simulation process that provides additional data sets. Method 500 proceeds at step 514 by training a machine learning algorithm using the simulated data set, the clear data set, and the multi-source sensing data set and derives a trained machine learning model. Then, at step 516, method 500 continues by using the trained machine learning model to infer new data and generate latent data. Method 500 then checks at verification 518 whether there is sufficient latent data for use. In a negative case, method 500 reverts to training the machine learning algorithm and takes into account the feedback information from the latent data. When there is sufficient data available, method 500 continues at step 520 by analyzing the data to obtain impact data. The impact data is then used to optimize real-world entities, with the goal of reducing the impact of real-world elements from the real-world system, as shown at step 522.

[0091] While certain embodiments are described and shown in the drawings, it should be understood that such embodiments are illustrative only and do not limit the broad invention, and since various other modifications will occur to those of ordinary skill in the art, the invention is not limited to the specific architectures and arrangements shown and described. Accordingly, the description is illustrative rather than restrictive.

Claims

1. A system for providing real-world data with multi-source real-time situational awareness to machine learning algorithms for artificial intelligence applications, the system comprising: A server computer system configured to store and process input data, the server computer system including one or more server computers, each including a memory and a processor; And A plurality of interconnected elements are interconnected with each other through a network and connected to the server, each connected element including a sensing mechanism configured to capture multi-source sensing data from real-world entities; Wherein the server computer system stores a persistent virtual world system that stores a virtual copy of the real-world entity; this system is updated according to the multi-source sensing data, and wherein the server computer system is configured to: Prepare data including explicit data and the multi-source sensing data to generate a sensing and explicit data set, wherein the explicit data set represents the expected functions and behaviors of the real-world entity; Train a machine learning algorithm using a machine learning input data set including the sensing data set and the explicit data set to generate a trained machine learning data set; Use the trained machine learning data set for inference on new data and derive implicit data to reflect the real behaviors of the real-world entity during the operation of the real-world entity; and Use the implicit data to further train the machine learning algorithm.

2. The system according to claim 1, wherein the server computer system is further configured to perform simulations, perform impact analysis by using impact data representing each virtual copy, execute multiple virtual copies during system operation, or optimize the real world using data obtained from the memory, the optimization including managing the copies corresponding to the persistent virtual world system.

3. The system according to claim 1, wherein the virtual copy is modeled by a copy editor stored in the memory in a manner that increases explicit data and instructions corresponding to each real-world entity.

4. The system according to claim 3, wherein the explicit data and instructions input through the copy editor describe the shape, location, position and orientation, physical properties, and the expected functions and impacts of each real-world entity.

5. The system according to claim 1, wherein the virtual copy is further updated according to user input, calculations of the server computer system, or a combination of both.

6. The system according to claim 1, wherein the multi-source sensing data includes data that can be captured for each of the real-world entities, including one or more of 3D image data, 3D geometry, 3D entities, 3D sensing data, 3D dynamic objects, video data, audio data, priority data, chemical composition, waste generation data, text data, time data, location data, orientation data, speed data, temperature data, humidity data, pollution data, light data, volume data, flow rate data, chromaticity data, energy consumption data, bandwidth data, or mass data.

7. The system according to claim 1, wherein the multi-source sensing data further includes situational data, the situational data including micro-situations that directly affect real-world elements, and macro-situations derived from multiple micro-situations, wherein the micro-situations include 3D graphic data, 3D geometry, 3D entities, 3D sensing data, 3D dynamic objects, video data, audio data, text data, time data, metadata, priority data, security data, location data, light data, temperature data, quality of service, or service situations that directly surround and affect the environment of a target real-world element.

8. The system according to claim 1, wherein the sensing mechanism includes one or more temperature sensors, proximity sensors, inertial sensors, infrared sensors, pollution sensors, pressure sensors, light sensors, ultrasonic sensors, smoke sensors, touch sensors, color sensors, humidity sensors, water sensors, or electronic sensors, or a combination thereof.

9. The system according to claim 1, wherein the connected elements include one or more mobile phones, laptops, wearable computers, mobile gaming devices, head-mounted displays, see-through devices, surveillance cameras, vehicles, traffic lights, buildings, streets, railway tracks, or household appliances.

10. The system according to claim 1, wherein the machine learning training and inference are implemented operationally or organizationally together with the target.

11. The system according to claim 1, wherein the machine learning training is performed by using the explicit data and an additional data set obtained by simulating the multi-source sensing data captured by the connected elements.

12. A method for providing multi-source, real-time, and situation-aware real-world elements to a machine learning algorithm for artificial intelligence applications includes: providing a server computer system including one or more server computers, each server computer including a processor and a memory, and the server computer system implementing a persistent virtual world system that stores multiple virtual copies of real-world entities; providing a plurality of connected elements connected to the server computer system via a network, each connected element including one or more sensing mechanisms; capturing, by the connected elements, multi-source sensing data from real-world entities, which can enrich and update the virtual copies, and transmitting the data to the server computer system; Prepare data by the server computer system, including explicit data and the multi-source sensing data, so as to derive a sensing and explicit data set, wherein the explicit data set represents the expected functions and behaviors of the real-world entities; Train a machine learning algorithm with a machine learning input data set containing the sensing and explicit data set by the server computer system, and derive a trained machine learning data set; Use the trained machine learning data set by the server computer system in an artificial intelligence application to manage the operation of the real-world entity and generate implicit data, where the implicit data reflects the real behaviors of the real-world entity during operation; and Further train the machine learning algorithm by the computer system using the implicit data.

13. The method according to claim 12, further comprising: Analyze the implicit data by the server computer system to obtain a representation of each virtual copy and the impacts caused by multiple virtual copies during system operation; and Optimize a real-world system including a real-world entity, and the optimization includes managing the corresponding copies of the persistent virtual world system.

14. The method according to claim 12, further comprising performing simulations of the sensing and explicit data set by the server computer system to derive an additional machine learning input data set.

15. The method according to claim 12, wherein the explicit data input through the copy editor describes the shape, location, position and orientation, physical properties, and expected functions and impacts of each of the real-world entities.

16. The method according to claim 12, wherein the multi-source sensing data includes one or more of 3D imaging data, 3D geometric data, 3D entities, 3D sensing data, 3D dynamic objects, video data, audio data, priority data, chemical composition, waste generation data, text data, time data, location data, orientation data, speed data, temperature data, humidity data, pollution data, light data, volume data, flow data, chromaticity data, energy consumption data, bandwidth data, or mass data.

17. The method according to claim 12, wherein the multi-source sensing data includes scenario data, and the scenario data includes micro-scenarios directly affecting a real-world element and macro-scenarios derived from multiple micro-scenarios, wherein the micro-scenarios include 3D graphic data, 3D geometric data, 3D entities, 3D sensing data, 3D dynamic objects, video data, audio data, text data, time data, metadata, priority data, security data, location data, light data, temperature data, quality of service, or the service scope directly surrounding and affecting the environment where the target real-world element is located.

18. The method according to claim 12, wherein the preparatory data includes attaching identification metrics to one or more virtual copies, wherein the identification metrics include descriptive information about the corresponding real-world elements, and the identification metrics are used to supply labeled data to a supervised training algorithm.

19. The method according to claim 12, wherein the training and inference of the machine learning algorithm are implemented operatively or organizationally together with a target.

20. The method according to claim 12, wherein the training of the machine learning algorithm is performed using an additional data set obtained by simulating the sensing data set and the explicit data set.

Images