Local computing cloud interacting with public computing cloud

Abstract

The invention relates to a local computing cloud interacting with a public computing cloud. A home computing cloud (HCC) may support one or more Internet of Things (IoT) devices using different connection protocols in a local environment. Typically, the HCC reduces the amount of data traffic sent to a public computing cloud (PCC) by processing collected device data locally rather than by sending the device data to the PCC for processing. This approach reduces the amount of data traffic sent over the network, improves data privacy, and helps maintain a desired level of quality of service. To do so, the HCC may download an appropriate data analysis model from the PCC, train the model, execute the trained model to obtain prediction information from collected IoT device data, and upload the trained model to the PCC. Alternatively, the HCC and PCC may execute sub-models of the analytical model and exchange outputs of the sub-models with each other.

Description

[0001] This application is a divisional application of the application with the application date of April 6, 2021, application number 202180035780.1, and invention title "Local Computing Cloud Interacting with Public Computing Cloud". Technical Field

[0002] Aspects of the present disclosure relate to a local computing cloud that supports interaction with a public computing cloud. The local computing cloud can be located in a home and can support one or more Internet of Things (IoT) devices. Analysis models can be downloaded from the public computing cloud and executed locally. Reinforcement training can also be performed locally without externally transmitting device data and user behavior information, thus greatly reducing the data traffic that may endanger data privacy. Background Art

[0003] Internet of Things (IoT) applications typically rely on remote and centralized servers to collect input data and generate certain actions based on the current input and historical data. This approach generally requires IoT devices such as smart sensors, thermostats, and smart appliances to exchange data between themselves and a remote server such as a public computing cloud. Using another approach, a gateway may be required to convert data from one connection protocol to another to send data from a terminal device to a server, e.g., from ZigBee to WiFi. The large amount of data transmission between the terminal device and the server means expensive service costs. In addition, this may generate a large amount of data traffic in the network, which may result in additional network latency, data loss during transmission, or expensive maintenance costs in order to maintain a desired quality of service level. Additionally, data security and privacy are an important concern when storing a large amount of personalized data in a public computing cloud. Summary of the Invention

[0004] A home computing system (which may be referred to as a "home computing cloud") integrates a communication gateway, a WiFi router, a cloud server, and a mass storage device to support one or more Internet of Things (IoT) devices in a local environment such as a home residence. Since the home computing cloud (HCC) locally processes the collected device data instead of sending the device data to a public computer cloud (system) for processing, the home computing cloud generally reduces the amount of data traffic sent to the public computing cloud (PCC). This approach improves network latency, reduces data loss during transmission, and helps maintain a desired quality of service level.

[0005] To do so, the HCC can download an appropriate data analysis model (which may be referred to as a "model" and selected from a plurality of data analysis models) from the PCC based on configuration information (e.g., the types of IoT devices supported). The HCC can then locally execute the model by obtaining device data from one or more IoT devices, applying some or all of the device data to the model, and obtaining a prediction result from the model. The prediction result can then be applied to one or more of the supported IoT devices to affect the operation of one or more IoT devices.

[0006] On the other hand, the HCC sends a subset of the device data to the PCC for further processing and receives decision information based on the data subset. For example, the subset of device data can represent one or more signal characteristics of a complex signal (e.g., a multimedia signal including voice, music, image, or video signals), and the one or more signal characteristics require intensive processing that the HCC may not be able to support. In an exemplary method for face recognition, the HCC can implement the image preprocessing layer and the feature extraction layer of the analysis model and send the resulting data to the PCC for analysis and decision-making. The HCC applies the received result and other device data (corresponding to the model input) to the downloaded data analysis model.

[0007] On the other hand, the PCC executes the input processing layer of the prediction model and sends the corresponding output to the PCC. The PCC then executes all the remaining hidden layers and sends the corresponding output of the final hidden layer back to the HCC. The HCC then executes the output layer. This approach generally eliminates sensitive information sent over the Internet and thus enhances data privacy on the Internet.

[0008] On the other hand, the distribution of the workload for executing the model can be based on the computing power of the HCC (such as sending the raw data to the PCC for the entire process); the amount of data traffic (such as sending only the feature data to the PCC to process the remaining tasks); data privacy (such as sending the mathematical transformation data within the model layer to the PCC to continue the analysis); the consistency of the model parameters (such as the HCC execution layer with fixed parameters and the PCC executable layer with parameters continuously changing via reinforcement training).

[0009] On the other hand, the HCC can have sufficient computing resources to perform more complex tasks, such as training a deep neural network. The HCC can download an appropriate template of the data analysis model from the PCC, train the model locally, and execute the trained model to obtain prediction information from the collected IoT device data.

[0010] On the other hand, both HCC and PCC can execute and train the same data analysis model (e.g., assist in training). However, the learning rates of HCC and PCC may be different (e.g., due to the stronger computing power of PCC). Although HCC executes and trains a local model based on IoT device data, HCC also sends the device data to PCC. PCC uses the same device data to execute and train, and sends the error measurement values back to HCC. HCC compares the error measurement values with the two clouds and continues training until the error measurement values from HCC are below the threshold.

[0011] On the other hand, when the error measurement values from PCC are continuously lower than or substantially lower than those of HCC, HCC can decide to continue training using the parameters from PCC.

[0012] On the other hand, if the error measurement values from PCC reach the threshold first, HCC can decide to use the model trained by PCC and stop training.

[0013] On the other hand, HCC can upload the trained model to PCC for archiving, sharing, or optimization.

[0014] On the other hand, PCC can analyze all the received models from other HCCs and optimize the new model. PCC can assign the new model to all HCCs.

[0015] On the other hand, HCC can decide to fully use the new model, use the new model with the parameters from the existing model, or completely ignore the new model. When executing different models using the locally stored empirical data, the decision can be based on the comparison with the error measurement values.

[0016] On the other hand, HCC can continue to run or train the local model, and PCC can train the new model in parallel with the new input data. HCC continuously sends the new input data to PCC to train the new model until the new model is accurate enough. HCC can then download the new model for use.

[0017] On the other hand, HCC can request PCC to continuously train the new model using the parameters in HCC.

[0018] On the other hand, in training the model, subjective weights can be applied when calculating the error based on the application scenario.

[0019] The present disclosure also includes the following aspects:

[0020] 1) A home computing system supporting at least one Internet of Things (IoT) device, the home computing system comprising:

[0021] A communication gateway configured to connect to the at least one IoT device;

[0022] A cloud interface configured to exchange information with a public computing cloud;

[0023] A processor for executing computer-executable instructions;

[0024] A memory storing the computer-executable instructions, which when executed by the processor cause the home computing system to:

[0025] Send configuration data about the home computing system, where the configuration data describes the home computing system;

[0026] Download a downloaded data analysis model based on the configuration data from the public computing cloud via the cloud interface;

[0027] Obtain device data from the at least one IoT device via the communication gateway;

[0028] Apply the device data to the downloaded data analysis model; and

[0029] Obtain a first prediction result from the data analysis model based on the device data.

[0030] 2) The home computing system according to 1), wherein the memory stores computer-executable instructions, which when executed by the processor further cause the home computing system to:

[0031] Determine a first error measurement value based on the first prediction result and correction information of the first prediction result;

[0032] When the first error measurement value exceeds a predetermined threshold, initiate reinforcement learning at the public computing cloud based on the device data, the first prediction result, and the correction information;

[0033] In response to the sending, receive modified model parameters from the public computing cloud; and

[0034] Apply the modified model parameters to the data analysis model.

[0035] 3) The home computing system according to 2), wherein the memory stores computer-executable instructions, which when executed by the processor further cause the home computing system to:

[0036] When the first error measurement value does not exceed the predetermined threshold, apply the prediction result to the at least one IoT device.

[0037] 4) The home computing system according to 1), wherein the memory stores computer-executable instructions that, when executed by the processor, further cause the home computing system to:

[0038] When a new IoT device is added to the home computing system, download a new data analysis model from the public computing cloud;

[0039] Apply current model parameters to the new data analysis model;

[0040] Initiate reinforcement learning for the new data analysis model;

[0041] In response to the initiation, obtain new model parameters for the new data analysis model; and

[0042] Re-apply the new model parameters to the new data analysis model.

[0043] 5) The home computing system according to 4), wherein the new model parameters are received from the public computing cloud.

[0044] 6) The home computing system according to 1), wherein the memory stores computer-executable instructions that, when executed by the processor, further cause the home computing system to:

[0045] Initiate reinforcement learning at the home computing system based on the device data, the first prediction result, and first correction information;

[0046] Obtain first updated model parameters from the reinforcement learning; and

[0047] Apply the first updated model parameters to the data analysis model to obtain an updated data analysis model.

[0048] 7) The home computing system according to 6), wherein the memory stores computer-executable instructions that, when executed by the processor, further cause the home computing system to:

[0049] Apply the device data to the updated data analysis model;

[0050] Obtain a second prediction result from the updated data analysis model based on the device data;

[0051] Determine a second error measurement based on the second prediction result and second correction information for the first prediction result;

[0052] When the second error measurement exceeds a desired error level, repeat the reinforcement learning;

[0053] Obtain a second updated model parameter from the repeated reinforcement learning; and

[0054] Re-apply the second updated model parameter to the data analysis model.

[0055] 8) The home computing system according to 7), wherein the memory stores computer-executable instructions that, when executed by the processor, further cause the home computing system to:

[0056] When the computing resources of the home computing system are exceeded, request cloud resources at the public computing cloud to execute the reinforcement learning.

[0057] 9) The home computing system according to 6), wherein the memory stores computer-executable instructions that, when executed by the processor, further cause the home computing system to:

[0058] Upload the updated data analysis model to the public computing cloud.

[0059] 10) The home computing system according to 1), wherein the memory stores computer-executable instructions that, when executed by the processor, further cause the home computing system to:

[0060] Execute the reinforcement learning of the downloaded data analysis model through the processor at the home computing system;

[0061] Obtain a first set of model parameters from the reinforcement learning at the home computing system;

[0062] Initiate auxiliary learning at the public computing cloud based on the device data, where the auxiliary learning is executed in parallel with the execution of the downloaded data analysis model at the home computing system;

[0063] Obtain a second set of model parameters from the auxiliary learning at the public computing cloud; and

[0064] Compare a third error measurement of the reinforcement learning with a fourth error measurement of the auxiliary learning.

[0065] 11) The home computing system according to 10), wherein the auxiliary learning utilizes a copy of the downloaded data analysis model.

[0066] 12) The home computing system according to 10), wherein the memory stores computer-executable instructions that, when executed by the processor, further cause the home computing system to perform the following operations:

[0067] When the third error measurement value is less than the fourth error measurement value, select the first set of model parameters from the first set of model parameters and the second set of model parameters; and

[0068] Apply the first set of model parameters to the data analysis model executed at the home computing system.

[0069] 13) The home computing system according to 10), wherein the memory stores computer-executable instructions that, when executed by the processor, further cause the home computing system to:

[0070] When the third error measurement value is greater than the fourth error measurement value, select the second set of model parameters from the first set of parameters and the second set of parameters; and

[0071] Apply the second set of model parameters to the data analysis model executed at the home computing system.

[0072] 14) The home computing system according to 10), wherein the memory stores computer-executable instructions that, when executed by the processor, further cause the home computing system to:

[0073] When one of the third error measurement value and the fourth error measurement value is less than a target error threshold, terminate the training of the data analysis model.

[0074] 15) The home computing system according to 14), wherein the memory stores computer-executable instructions that, when executed by the processor, further cause the home computing system to:

[0075] In response to the termination, upload the data analysis model to the public computing cloud. 16) A method for supporting a home computing system, the method comprising:

[0076] Obtain device data from at least one IoT device configured at the home computing system by the home computing system;

[0077] Stream the device data from the home computing system to the public computing cloud;

[0078] Provide the device data from the public computing cloud to a data analysis model that the public computing cloud is executing for the home computing system;

[0079] In response to the provision, obtain a first prediction result by the public computing cloud;

[0080] In response to the streaming, receive the first prediction result from the public computing cloud by the home computing system;

[0081] The first prediction result is applied to the at least one IoT device by the home computing system;

[0082] The home computing system determines an error measurement value based on the first prediction result and correction information of the first prediction result;

[0083] When the error measurement value exceeds a predetermined threshold, the home computing system sends the correction information to the public computing cloud;

[0084] The public computing cloud performs reinforcement learning based on the received correction information to obtain updated model parameters; and

[0085] The public computing cloud applies the updated model parameters to the data analysis model.

[0086] 17) The method according to 16), comprising:

[0087] In response to the sending of the correction information, the home computing system continues to stream the device data to the public computing cloud;

[0088] In response to the continued streaming, the home computing system receives a second prediction result from the public computing cloud; and

[0089] The second prediction result is applied to the at least one IoT device.

[0090] 18) A method for supporting a home computing system, the method comprising:

[0091] Identifying a data analysis model of the home computing system, wherein the home computing system supports at least one IoT device, and wherein the data analysis model comprises a set of layers;

[0092] Dividing the data analysis model into a plurality of sub-models, wherein each sub-model comprises a different subset of the layers of the data analysis model, and wherein the plurality of sub-models comprises a first sub-model and a second sub-model;

[0093] The home computing system executes the first sub-model; and

[0094] The public computing cloud executes the second sub-model.

[0095] 19) The method according to 18), wherein the first sub-model comprises an input processing layer, and the second sub-model comprises all hidden layers and an output layer.

[0096] 20) The method according to 18), wherein the plurality of sub-models further includes a third sub-model, and the method includes:

[0097] Executing the third sub-model by the home computing system, wherein the first sub-model includes an input layer, the second sub-model includes all hidden layers, and the third sub-model includes an output layer. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] The above summary of the present invention and the following detailed description of the exemplary embodiments of the present invention will be better understood when read in conjunction with the accompanying drawings, which are included by way of example and not as a limitation of the claimed invention.

[0099] Figure 1 Shows a home environment in which a Home Computing Cloud (HCC) interacts with a Public Computing Cloud (PCC) according to an embodiment.

[0100] Figure 2 Shows an HCC without WiFi router capabilities according to an embodiment.

[0101] Figure 3 Shows an HCC with WiFi router capabilities according to an embodiment.

[0102] Figure 4 Shows a PCC interacting with multiple HCCs according to an embodiment.

[0103] Figure 5 Shows an HCC interacting with a PCC and a user application according to an embodiment.

[0104] Figure 6 Shows an HCC according to an embodiment, where the HCC is executing an analysis model while the PCC is executing reinforcement training.

[0105] Figure 7 Shows an HCC according to an embodiment, where the HCC assigns all data analysis and reinforcement training tasks to the PCC.

[0106] Figure 8 Shows a method for an HCC to divide an analysis model into two sub-models according to an embodiment. A part of the original model is executed at the HCC, and the remaining part of the model is executed at the PCC to reduce the computation at the HCC, reduce data traffic, and protect data privacy when sending data over the network.

[0107] Figure 9 Shows a method for an HCC to perform reinforcement learning according to an embodiment.

[0108] Figure 10Shows a method for HCC and PCC to interact to perform assisted learning according to an embodiment. Detailed Description

[0109] "HCC" (Home Computing Cloud) may not be limited to a home residence and may support other types of entities such as enterprises or buildings. Thus, "HCC" can be understood as "Local Computing Cloud". Also, the "cloud" can be referred to as a computing system etc.

[0110] According to aspects of an embodiment, HCC integrates a communication gateway, a WiFi router, a cloud server, and a mass storage device to support one or more Internet of Things (IoT) devices in a local environment such as a home residence. Since HCC processes the collected device data locally instead of sending the device data to a Public Computing Cloud (PCC) for processing, HCC generally reduces the amount of data traffic sent to the PCC. This approach improves network latency, reduces data loss during transmission, and helps maintain a desired quality of service level. To do so, HCC may download an appropriate data analysis model (which may be referred to as a "model") from the PCC based on configuration information (e.g., the types of IoT devices supported). Then, HCC may execute the model locally by obtaining device data from one or more IoT devices, applying some or all of the device data to the model, and obtaining a prediction result from the model. The prediction result may then be applied to one or more of the supported IoT devices to affect the operation of one or more IoT devices.

[0111] HCC may include one or more IoT devices located in a home. Embodiments support various IoT devices including but not limited to smart thermostats, appliances, lighting devices, security devices, etc.

[0112] HCC may interact with the PCC to exchange information related to one or more IoT devices. The information may include data provided by one or more IoT devices (e.g., temperature measurements) and information indicating actions performed by one or more IoT devices (e.g., operating modes).

[0113] PCC (which may be referred to as the "Public Cloud") may provide computing services offered by a third-party provider over the public Internet, enabling anyone who wants to use or purchase these services to do so. The services may be free or sold on demand, thus allowing customers to pay only for usage in consumed CPU cycles, storage, or bandwidth.

[0114] In another aspect of the embodiments, the algorithm can be used to locally train a data analysis model. Reinforcement (machine) learning can also be added to provide machine learning capabilities to the HCC. By processing data locally, the user's privacy can be substantially improved by limiting the amount and type of data sent over the network and stored in the PCC.

[0115] In another aspect of the embodiments, the HCC (Home Computing System) performs both the data analysis model and reinforcement learning locally.

[0116] In another aspect of the embodiments, the data analysis model is divided into two sub-models. The first sub-model includes the input processing layer of the data analysis model and is executed by the home computing system (cloud). The second sub-model includes the hidden layer of the data analysis model and is executed by a public computing cloud. By this method, the original data is locally stored in the computing system, thus protecting the user's privacy.

[0117] In another aspect of the embodiments, the data analysis model is divided into three sub-models. The first sub-model and the third sub-model include the input layer and the output layer respectively, and are executed by the home computing system (cloud). The second sub-model includes only the hidden layer and is executed by a public computing cloud. By this method, the user's privacy is further protected by locally maintaining the original input data as well as the predicted output.

[0118] In another aspect of the embodiments, assisted learning enables training to be performed in a public computing cloud while the data analysis model is being executed at the home computing system.

[0119] In another aspect of the embodiments, assisted learning enables parallel training to be performed in both the public computing cloud and the home computing cloud while the data analysis model is being executed at the home computing system.

[0120] Figure 1 A home environment is shown in which the HCC 101 interacts with the PCC 102 via a data channel 151 according to an embodiment.

[0121] IoT devices (not explicitly shown) can be interconnected computing devices (e.g., smart thermostats or appliances) within the home that provide information to be sent and receive information via the HCC 101. The received information can indicate one or more actions that the IoT device should perform.

[0122] Although Figure 1 an operating environment spanning a home is depicted, the embodiments can span other local environments such as buildings or commercial premises.

[0123] When implementing an Internet of Things (IoT) system, data traffic capacity, data security, and data privacy are important considerations. By minimizing data traffic and carefully selecting the type of data to be sent on the data channel 151 between the application environment supported by PCC 102 and HCC101, and by minimizing the amount and type of data stored inside PCC 102, data exposure due to unauthorized access can be reduced. In addition, data traffic can be reduced, and the cost of using services provided by PCC 102 can therefore be reduced. By storing data in HCC 101 and performing data analysis and machine learning from HCC 101, services can be maintained when an Internet connection is inaccessible. In addition, delays introduced from an Internet connection can be eliminated. However, since PCC 102 typically provides computing power and software services that HCC 101 may not provide, it may not be possible to completely circumvent the services provided by the PCC.

[0124] When deciding which data and functions to store locally in the HCC 101 and which data and services to allocate to the PCC 102, one approach is to store discrete time interval data (e.g., sensor data, manual settings, etc.) together with machine learning algorithms and data analysis models in the HCC 101, as will be discussed in further detail. The HCC 101 can continuously send supported IoT devices (e.g., such as the HCC 102) to the PCC 102 via the data channel 151. Figure 2 HCC 101 may also periodically send information about the training model (eg, parameters and error measurements of the analysis model) to public computing cloud 102.

[0125] PCC 102 can collect data from all available HCCs 101 and 401 (such as Figure 4 Further discussion), and train a new pre-trained model, such as a model template, based on the collected data. The new pre-trained model can then be distributed back to each HCC 101 and 401. Alternatively, PCC 102 can notify HCC 101 and / or 401 that a new pre-trained model is available, wherein HCC 101 and / or 401 can decide whether to download the model via data channel 151 based on predetermined criteria.

[0126] As will be discussed in further detail, some or all of the data from the previous model can be directly applied to the model template. Alternatively, if no data is available, reinforcement learning can be applied using the model template with locally stored model data or completely new data.

[0127] As will be discussed further in detail, parallel training (machine learning in both the home and the PCC) can be applied when performing reinforcement learning. Model parameters can be exchanged during training. The model adopted by the HCC can be selected based on the error measurement value.

[0128] The PCC 102 can consistently update the machine learning algorithm to the HCC 101.

[0129] For continuous-time signals (e.g., audio, image, and video data), the HCC 101 can stream the signal data to the PCC 102. When a data analysis model is supported at the PCC 102, the results from the model can be returned to the HCC 101.

[0130] Alternatively, the analysis model can be split into two parts (e.g., sub-models) and partially executed at the HCC and the PCC. The data exchange between the two clouds can be the parameters in one or more layers of the analysis model. This approach can reduce the amount of data to be exchanged between the two clouds. Additionally, privacy can be maintained relative to sending the original data stream.

[0131] Alternatively, the analysis model can be split into three sub-models and partially executed at the HCC and the PCC. In this way, the input processing layer and the output layer of the analysis model are executed at the HCC, and the hidden layer is executed at the PCC. In this manner, the original input data and the predicted output, which may contain private information about the device owner, will be kept locally and not disclosed externally.

[0132] Alternatively, the model can be trained in the PCC 102, downloaded from the PCC 102 to the HCC 101, and executed locally by the HCC 101. The decision to retrain a new model can be triggered by the owner (user). Examples include adding a new device for identification to the model, adding new rules to the model, etc.

[0133] Figure 2 Shown is an HCC 201 associated with a separate WiFi router 206 according to an embodiment, which interacts with a PCC 202.

[0134] The interaction between local IoT devices 204-205 can be supported by the protocol gateway 210 and the IoT message converter 211 executed at the HCC 201. The IoT devices 204-205 communicate via the protocol gateway 210 using the corresponding protocol (e.g., Zigbee). The protocol gateway 210 passes the device messages to the IoT message converter 211, which includes an IoT protocol message broker 208 (e.g., MQTT broker) or a COAP server (not explicitly shown) and an IoT protocol message bridge 209 (e.g., MQTT / Zigbee bridge). The message converter 211 bridges the IoT device messages into IoT protocol messages (e.g., MQTT messages). The MQTT messages can be directed to other IoT devices connected to the HCC 201, the rules engine, or the PCC 202. As an example, device messages from the Zigbee device 205 can be sent to the HCC 201 via the Zigbee gateway 210. The device data can be extracted from the device messages and sent to the analysis model for processing. At the same time, the device messages can be passed to the MQTT / Zigbee bridge 209 and the MQTT broker 208 to reach the PCC 202 via the home WiFi router 206.

[0135] WiFi devices (e.g., device 203) that support MQTT clients can also connect to the MQTT broker 208 within the HCC 201.

[0136] The device data collected by the HCC 201 can be stored in a mass data storage device (not explicitly shown), thus avoiding the additional cost of sending the collected data back and forth with the PCC 201.

[0137] The communication between the WiFi device 203 and the HCC 201 can occur through two different paths 251 or 252, depending on which WiFi access point the WiFi device 203 is connected to. For path 251, the MQTT messages from the WiFi device 203 are routed from the home WiFi router 206 to the MQTT broker 208, which can further direct the MQTT messages to other IoT devices or the PCC 202 via the home WiFi router 206. Through path 252, the WiFi device 203 is directly connected to the HCC 201, which acts as a WiFi access point (AP) and can also be connected to the home WiFi router 206 and then to the PCC 202.

[0138] The user application (app) 207 can interact with the HCC 201 and / or the PCC 202 via the WiFi connection 253.

[0139] Figure 3Shows an HCC 301 with WiFi router capabilities according to an embodiment. Since the HCC 301 includes a WiFi router 306, all WiFi devices (e.g., device 303) can connect to the HCC to access user applications 307 and / or the Internet services of the PCC 302. In addition, the adjacent mobile device 307 can also connect to the HCC 301 to access Internet services.

[0140] Figure 4 Shows a PCC 102 that interacts with multiple HCCs including an HCC 101 and an HCC 401. Thus, the PCC 102 can obtain data on data analysis models executed on multiple HCCs and can train the mirror model executed on the PCC 102. The PCC 102 can then allocate the trained model to one or more HCCs such that the trained model can be executed locally.

[0141] Figure 5 Shows an HCC 501 that interacts with a PCC 514 and user applications 512 according to an embodiment.

[0142] Similar to Figure 2 and Figure 3 , the HCC 501 interacts with IoT devices 504 - 506 via a communication server 507, the PCC 514 via a cloud interface 503, and a mobile device 512.

[0143] The HCC 501 includes a processing device 502, a cloud interface 503, a communication server 507, a memory device 509, and a storage device 511. Additionally, the HCC 501 can include an embedded WiFi router 508 in some embodiments (e.g., as Figure 3 shown).

[0144] The processing device 502 controls the operation of the HCC 501 by executing computer - readable instructions stored on the memory device 509. For example, the processing device 502 can execute computer - readable instructions to perform the processes 600 - 1000 shown in Figures 6 to 10 respectively. Embodiments can support various computer - readable media, which can be any available media accessible by the processing device 502 and include volatile and non - volatile media, removable and non - removable media. By way of example and not limitation, computer - readable media can include a combination of computer storage media and communication media.

[0145] A computer storage medium can include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic tape cartridges, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the required information and can be accessed by a computing device.

[0146] A communication medium typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transmission mechanism, and includes any information delivery medium. A modulated data signal is a signal whose one or more characteristics are set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media.

[0147] HCC 501 can execute the downloaded model 510 from PCC 514 at the memory device 509. The machine learning model 510 can include neural network models from IoT devices 504 - 506 processing data as input, thereby generating one or more decision outputs from the model 510.

[0148] If there are any corrective actions on the predicted output from the model, the home computing cloud 501 can apply reinforcement learning to train the model.

[0149] Figure 6 A logic flow 600 for locally executing an analysis model is shown. When HCC 101 is setting up or adding a new device, HCC 101 sends system configuration information to PCC 102 at block 601 and downloads the corresponding analysis model at block 602. The analysis model is implemented at block 603 and executed at block 604 based on IoT device input 651 and model parameters 650.

[0150] As an example, HCC 101 currently supports thermostats and presence sensors in a home residence. The thermostat learns that when there is a user at home from April to October, the operating mode should be set to cooling and the temperature should be set to 23°C, while from November to March, the operating mode should be set to heating and the temperature should be set to 25°C:

[0151] April - October: Presence [0: Mode = Off | 1: Mode = Cooling, Set_Temperature = 23].

[0152] November to March: There is [0: Mode = Off | 1: Mode = Heating, Set_Temperature = 25].

[0153] Continuing with the example, when the user adds a smart curtain (new IoT device) to the ecosystem, HCC101 sends configuration information (e.g., a profile) about the thermostat, presence sensor, and smart curtain to PCC 102. When adding the new device (smart curtain), PCC 102 notifies HCC 101 that a new analysis model is available. Since the setting "Use new model template" is set to "Yes", HCC 101 downloads the new model template. For the new model, the original settings are applied. Additionally, a new input parameter (IoT device input) "Degree of curtain opening (0% fully open to 100% fully closed)" is introduced. With the new model provided by PCC 102, the degree of curtain opening has no effect on the set temperature:

[0154] April to October: There is [0: Mode = Off | 1: Mode = Cooling, Set_Temperature = Curtain[0, 100:23]].

[0155] November to March: There is [0: Mode = Off | 1: Mode = Heating, Set_Temperature = Curtain[0, 100:26]].

[0156] During execution, HCC 101 continuously receives inputs from IoT device 651 and processes them through analysis model 604 to obtain prediction results. The prediction results are applied to the corresponding IoT device at block 605.

[0157] HCC 101 continuously monitors the IoT ecosystem to make any corrections to the prediction results at block 606. If any corrections are made, at block 607, HCC 101 provides the correction T[n]654 as feedback, along with the corresponding IoT device input S[n]652 and prediction result R[n]653, to PCC102, where the result information may include R[n]653 and T[n]654. For example, HCC 101 can conditionally initiate reinforcement learning at PCC 102 and thus receive updated parameters as a response. HCC can then update the downloaded analysis model.

[0158] Continuing with the above example, in July, when the smart curtain is approximately half-closed (e.g., curtain = 40% closed), the user changes the set temperature to 24°C:

[0159] 20200701: There is [1: Mode = Cooling, Set_Temperature = Curtain[40:24]].

[0160] When the curtain is slightly closed (e.g., curtain = 20% closed), the original set temperature remains unchanged (i.e., 23 °C), and thus there is no message received from HCC 101.

[0161] In this case, as long as available, HCC 101 continuously sends user corrections to PCC 102:

[0162] 20190706: There is [1: Mode = Cooling, Set_Temperature = curtain[40:24]].

[0163] 20190707: There is [1: Mode = Cooling, Set_Temperature = curtain[50:24]].

[0164] 20200713: There is [1: Mode = Cooling, Set_Temperature = curtain[65:24]].

[0165] 20200714: There is [1: Mode = Cooling, Set_Temperature = curtain[45∶24]].

[0166] 20200719: There is [1: Mode = Cooling, Set_Temperature = curtain[60:24]].

[0167] 20200720: There is [1: Mode = Cooling, Set_Temperature = curtain[40:24]].

[0168] 20200721: There is [1: Mode = Cooling, Set_Temperature = curtain[50∶24]].

[0169] At PCC 102, reinforcement learning is performed at block 621 to obtain a new set of model parameters (replacing model parameter 650). The new parameters 655 are sent to HCC 101 at block 622 and are used by the analysis model for the next device input S[n+1] at block 608.

[0170] Continuing with the above example, when the reinforcement training is complete, PCC 102 sends the new model parameters to HCC101. HCC 101 then applies the new parameters in the model. The new parameters are:

[0171] From April to October, there is [0: Mode = Off | 1: Mode = Cooling, Set_Temperature = curtain[0,40:23 | 40,100:24]].

[0172] From November to March, there is [0: Mode = Off | 1: Mode = Heating, Set_Temperature = curtain [0, 100: 26]].

[0173] Figure 7 A process 700 for processing multimedia signals is shown, where the analysis model is executed at PCC 102. At block 701, HCC 101 continuously streams source data 751 to PCC 102. At block 721, PCC 102 uses a model with Z hidden layers to analyze the data stream to obtain a prediction result R 752. Then at block 722, the result is sent back to HCC101, where at block 702, HCC 101 applies the prediction result to the IoT ecosystem.

[0174] HCC 101 continuously monitors the IoT ecosystem to make any corrections to the prediction result at block 703. If there are any corrections, then at block 704, HCC 101 provides a correction T753 to PCC 102 as feedback.

[0175] At block 723, PCC 102 performs reinforcement learning to obtain a new set of parameters W 754. Then at block 721, the new parameters are applied to the analysis model for subsequent source data streams.

[0176] Figure 8 An analysis model with Z hidden layers 801 is shown, which can be divided into two sub-models 804, one sub-model having X hidden layers 802 (implemented at HCC 101), and the other sub-model having Y hidden layers 803 (implemented at PCC102), where X + Y = Z. In application 800, source data streams from one or more IoT devices are analyzed at HCC 101. The output of the x-th hidden layer from hidden layer 802 is then sent to PCC 102 to continue the analysis through hidden layer 803.

[0177] Using method 800, the amount of data sent from HCC 101 to PCC 102 is generally reduced. In addition, the privacy of user data can be protected by sending a transformed version of the data rather than the source data.

[0178] In addition, the distribution of the workload for executing the model can be based on:

[0179] - The computing power ε of the home computing cloud. (For example, HCC 101 executes the first layer of the analysis model and then sends the output to PCC 102. PCC 102 then executes the remaining layers of the analysis model.

[0180] - The amount of data traffic. (For example, HCC 101 executes the layers of the analysis model until it reaches the layer with the minimum number of output nodes. HCC will then send the output of that layer to PCC 102. PCC 102 then executes the remaining layers of the analysis model).

[0181] - Data privacy. For example, HCC 101 can execute the layers of the analysis model until it reaches a layer position where the output of that layer is completely independent of the source data. HCC 101 then sends the output of that layer to PCC 102. PCC 102 then executes the remaining layers of the analysis model.

[0182] - Consistency of model parameters. For example, HCC 101 executes the layers of the analysis model with fixed parameters. HCC 101 then sends the output of the last layer to PCC 102. PCC 102 then executes the remaining layers of the analysis model).

[0183] The capabilities of HCC 101 can vary from those with only basic configurations that only allow the execution of the analysis model to more powerful capabilities equipped with more powerful hardware for training analysis models with multiple hidden layers.

[0184] In the case of further improving data privacy using another specific implementation, the analysis model can be divided into three sub-models. In this case, the input processing layer and the output layer are executed at HCC 101, and some or all of the hidden layers are executed at PCC 102. In this case, the original input data and the predicted output that are closely related to the user will be saved locally.

[0185] Figure 9 The logical flow 900 for locally executing reinforcement learning at HCC 101 is shown. Reinforcement learning is executed at block 901. HCC 101 uses the input data S[n] 951, the predicted output R[n] 952, and the correction T[n] 953 at the nth operation to optimize the parameter set W of the analysis model by minimizing the error function.

[0186] Upon completion of training, the new parameter set W[n+1] 954 is provided to the analysis model at block 902 so that the analysis model can utilize them at block 903. When there is new device input S[n+1] 955 from one or more IoT devices, the new predicted result R[n+1] 956 can be applied to the IoT ecosystem at block 904.

[0187] If there is a correction from the user at block 905, then at block 906, the reinforcement learning algorithm repeats the execution using the data (S, R, and T) from the [n+1] example. Otherwise, HCC 101 waits for input at block 907.

[0188] For some applications, training of the analysis model may be too resource - intensive for the computer resources of HCC 101. In such cases, PCC 102 can be used to assist with reinforcement learning at HCC 101.

[0189] Figure 10 A logic flow 1000 for assisting with training (parallel training) is shown, where HCC 101 performs training during a training sequence 1001, and PCC 102 performs parallel training during a training sequence 1021.

[0190] At block 1002, HCC 101 performs reinforcement learning using a device input S[n] 1051 and an analysis model G. At block 1003, the predicted output O[m] 1053 from the analysis model G is compared with a correction T[n] 1052 from the user to calculate an error measurement E[m] 1054. At block 1004, an adjustment to the model parameters is determined based on the magnitude of the error value and the rate of change of the error value between iterations. Then, at block 1002, the analysis model G uses a new parameter set U[m + 1] 1055 to calculate a new output O[m + 1] using the same device input S[n]. Then a new error measurement E[m + 1] is calculated by comparing T[n] with O[m + 1]. Additional iterations can be performed until a desired error measurement is obtained.

[0191] When performing reinforcement learning at HCC 101, a copy of the device input S[n] 1051 and the correction T[n] 1052 are sent to PCC 102, for example via a data channel 151. PCC 102 performs a similar reinforcement learning process at block 1021 to assist with model training at HCC 1001. Correspondingly, at block 1022, the device input S[n] 1051 is performed by an analysis model P. At block 1023, the predicted output Q[k] 1073 from the analysis model P at block 1022 is compared with the correction T[n] 1052 from the user to calculate an error measurement F[k] 1074. At block 1024, the adjustment to the model parameters is based on the magnitude of the error value and the rate of change of the error value between iterations. Then, the analysis model P 1022 uses a new parameter set V[k + 1] 1075 to calculate a new output Q[k + 1] using the same device input S[n], and a new error measurement F[k + 1] is calculated by comparing T[n] with Q[k + 1] and so on.

[0192] PCC 102 can use the same algorithm (where model P is a copy of model G) to change the model parameters U 1055 and V 1075. Alternatively, different algorithms (where model P is not a copy of model G) can be used to adjust the model parameters U 1055 and V 1075.

[0193] During auxiliary training, the error measurements from two learning models (G at block 1002 and P at block 1022) can be compared consistently. If there is a significant difference between the two error measurements, both learning models G and P can select (switch to) the model parameter set that produces the lower error measurement and continue training.

[0194] If either of the two models G and P meets the target error threshold, the training can be terminated, and the model parameter set that meets the error threshold is used by the analysis model at HCC 101.

[0195] At the end of reinforcement learning, HCC 101 can upload the trained model to PCC 102 for archiving, sharing, or optimization.

[0196] PCC 102 can analyze all models received from other HCCs and optimize new models from them. For some embodiments, when adding new default IoT devices, new models can be trained. PCC 102 can distribute the new models to all HCCs.

[0197] HCC 101 can decide to use the new model provided by PCC 102, use the new model with parameters from the original model, or completely ignore the new model. When executing different models using locally stored empirical data, the decision can be based on a comparison with the error measurements.

[0198] HCC 101 can decide to perform reinforcement learning at any time during operation, for example, according to block 621 (as Figure 6 shown), block 723 (as Figure 7 shown), block 906 (as Figure 9 shown), or process 1000 (as Figure 10 shown).

[0199] In reinforcement learning, traditional IoT device data locally stored at HCC 101 can be used to train the analysis model (e.g., for continuous improvement of the original model). Alternatively, new IoT device data can be used to train the analysis model (e.g., a new analysis model with additional device types). Alternatively, a mixture of traditional IoT device data and new IoT device data can be used to execute the original model and train a new model in parallel.

[0200] During reinforcement learning, weights can be assigned when calculating the error measurements. For example, for object recognition, more weight may be assigned to the recognition error than to the confidence level error.

[0201] Aspects described herein may be embodied as a method, an apparatus, or computer-executable instructions stored on one or more non-transitory and / or tangible computer-readable media. Accordingly, these aspects may take the form of an entirely hardware implementation, an entirely software implementation (which may include or may not include firmware) stored on one or more non-transitory and / or tangible computer-readable media, or an implementation combining software and hardware aspects. Any and / or all of the method steps described herein may be embodied in computer-executable instructions stored on a computer-readable medium, such as a non-transitory and / or tangible computer-readable medium and / or a computer-readable storage medium. Additionally or alternatively, any and / or all of the method steps described herein may be embodied in computer-readable instructions stored in a memory and / or other non-transitory and / or tangible storage medium of an apparatus including one or more processors such that, when the computer-readable instructions are executed by the one or more processors, the apparatus performs such method steps. Additionally, various signals representing data or events described herein may be transmitted between a source and a destination in the form of light waves and / or electromagnetic waves that travel through a signal-conducting medium, such as metal wires, optical fibers, and / or wireless transmission media (e.g., air and / or space).

[0202] Aspects of the present disclosure have been described in terms of its illustrative embodiments. Many other embodiments, modifications, and variations within the scope and spirit of the appended claims will occur to those of ordinary skill in the art upon viewing the present disclosure. For example, those skilled in the art will appreciate that the steps illustrated in the illustrative figures may be performed in an order other than the recited order, and that one or more of the steps illustrated may be optional in accordance with aspects of the present disclosure.

Claims

1. A method for supporting a home computing system, the method comprising: obtaining, by the home computing system, device data from at least one Internet of Things (IoT) device configured at the home computing system; streaming, by the home computing system, the device data to a public computing cloud; providing, by the public computing cloud, the device data to a data analysis model that the public computing cloud is executing for the home computing system; obtaining, by the public computing cloud in response to the providing, a first prediction result; receiving, by the home computing system in response to the streaming, the first prediction result from the public computing cloud; applying, by the home computing system, the first prediction result to the at least one IoT device; determining, by the home computing system, an error measurement value based on the first prediction result and correction information of the first prediction result; when the error measurement value exceeds a predetermined threshold, sending, by the home computing system, the correction information to the public computing cloud; performing, by the public computing cloud, reinforcement learning based on the received correction information to obtain updated model parameters; and applying, by the public computing cloud, the updated model parameters to the data analysis model.

2. The method according to claim 1, further comprising: continuing to stream, by the home computing system, the device data to the public computing cloud in response to the sending of the correction information; receiving, by the home computing system in response to the continued streaming, a second prediction result from the public computing cloud; and applying the second prediction result to the at least one IoT device.

3. The method according to claim 1, further comprising: when the error measurement value does not exceed the predetermined threshold, continuing to stream, by the home computing system, the device data to the public computing cloud without sending correction data.

4. A method for supporting a home computing system, the method comprising: identifying a data analysis model of the home computing system, wherein the home computing system supports at least one Internet of Things (IoT) device, and wherein the data analysis model comprises a set of layers; dividing the data analysis model into a plurality of sub-models, wherein each sub-model comprises a different subset of the set of layers of the data analysis model, and wherein the plurality of sub-models comprises a first sub-model and a second sub-model, the first sub-model comprises a first subset of the set of layers, and the second sub-model comprises a second subset of the set of layers; executing, by the home computing system, the first sub-model using an input based on source data of the at least one IoT device; sending, based on the execution of the first sub-model, output data from the first sub-model from the home computing system to a public computing cloud; and executing, by the public computing cloud, the second sub-model using an input based on the output data from the home computing system.

5. The method according to claim 4, wherein the first sub-model includes an input processing layer of the data analysis model, and the second sub-model includes one or more hidden layers of the data analysis model and an output layer of the data analysis model.

6. The method according to claim 4, wherein the plurality of sub-models further includes a third sub-model, and the method further includes: executing the third sub-model by the home computing system, wherein the first sub-model includes an input layer of the data analysis model, the second sub-model includes only the hidden layers of the data analysis model, and the third sub-model includes an output layer of the data analysis model.

7. The method according to claim 4, wherein The plurality of sub-models further includes a third sub-model, and the method further includes: sending second output data from the second sub-model from the public computing cloud to the home computing system; and executing the third sub-model by the home computing system based on the second output data from the second sub-model.

8. The method according to claim 7, wherein The first sub-model includes an input layer of the data analysis model, the second sub-model includes all hidden layers, and the third sub-model includes an output layer of the data analysis model.

9. The method according to claim 4, wherein Executing the first sub-model by the home computing system includes: executing a set of consecutive layers of the data analysis model from an input layer of the data analysis model to a predetermined layer of the data analysis model, and the method further includes: sending, by the home computing system, output data from the predetermined layer to the public computing cloud; wherein executing the second sub-model includes: executing, by the public computing cloud, the remaining layers of the data analysis model using the output data received from the home computing system as an initial input.

10. The method according to claim 9, wherein, Executing the first sub-model includes inputting source data from the at least one IoT device into the input layer, wherein the source data includes user data associated with a user of the at least one IoT device; and wherein the output data is transformed so as not to include the user data in the source data.

11. The method according to claim 4, the method further includes: determining a predetermined amount of generated traffic data between the home computing system and the public computing cloud; wherein executing the first sub-model by the home computing system includes executing layers of the data analysis model until a layer of the data analysis model having a minimum number of nodes, wherein the minimum number of nodes is based on the predetermined amount of generated traffic data between the home computing system and the public computing cloud.

12. The method according to claim 4, wherein, Executing the first sub-model by the home computing system includes: inputting source data from the at least one IoT device into an input layer of the data analysis model, the source data containing private information associated with a user of the at least one IoT device; and executing the data analysis model until a first layer with output data that does not include the private information of the source data is reached; wherein the method further includes sending the output data to the public computing cloud; and Among them, executing the second sub-model includes the public computing cloud using the output data as an initial input to execute the remaining layers of the data analysis model.

13. The method according to claim 9, the method further comprising: executing, by the home computing system, a set of layers, the set of layers including all layers of the data analysis model having fixed parameters; and sending, by the home computing system, output data from the last layer in the set of layers having the fixed parameters to the public computing cloud; and executing, by the public computing cloud, the remaining layers of the data analysis model.

14. The method according to claim 4, wherein The data analysis model includes a deep neural network.

15. A home computing system, the home computing system supporting at least one Internet of Things (IoT) device, the home computing system comprising: a communication gateway configured to connect to the at least one IoT device; a cloud interface configured to exchange information with a public computing cloud; a processor for executing computer-executable instructions; a memory storing the computer-executable instructions, the computer-executable instructions, when executed by the processor, causing the home computing system to: obtain device data from at least one IoT device configured at the home computing system via the communication gateway; stream the device data to the public computing cloud via the cloud interface; the public computing cloud providing the device data to a data analysis model that the public computing cloud is executing for the home computing system; receiving a first prediction result from the public computing cloud in response to the streaming; applying the first prediction result to the at least one IoT device; determining an error measurement based on the first prediction result and correction information of the first prediction result; and when the error measurement exceeds a predetermined threshold, sending the correction information to the public computing cloud.

16. The home computing system according to claim 15, wherein, The memory stores the computer-executable instructions, the computer-executable instructions, when executed by the processor, further causing the home computing system to: continue streaming the device data to the public computing cloud in response to sending the correction information; receiving a second prediction result from the public computing cloud in response to the continued streaming; and applying the second prediction result to the at least one IoT device.

17. A home computing system, the home computing system supporting at least one Internet of Things (IoT) device, the home computing system comprising: a communication gateway configured to connect to the at least one IoT device; a cloud interface configured to exchange information with a public computing cloud; a processor for executing computer-executable instructions; a memory storing the computer-executable instructions, the computer-executable instructions, when executed by the processor, causing the home computing system to: obtain a source data stream from one or more IoT devices from the communication gateway; Execute a first sub - model of a data analysis model using the source data stream as input, where the data analysis model includes a set of layers, and the set of layers includes: A first subset of the set of layers, which constitutes the first sub - model of the data analysis model; and A second subset of the set of layers, which is different from the first subset and constitutes the second sub - model of the data analysis model; Obtain transformed data from the first sub - model of the data analysis model that uses the source data stream as input; Send the transformed data to the public computing cloud via the cloud interface, where the public computing cloud uses the transformed data as input to execute the second sub - model of the data analysis model; In response to the sending, receive output data from the public computing cloud, where the output data is provided via the execution of the second sub - model of the data analysis model using the transformed data as input; Determine a prediction result from the output data; and Apply the prediction result to the at least one IoT device.

18. The home computing system according to claim 17, wherein, The first sub - model includes an input processing layer.

19. The home computing system according to claim 18, wherein, The set of layers further includes a third subset, which is different from the first subset and the second subset and constitutes a third sub - model, where the computer - executable instructions, when executed by the processor, further cause the home computing system to: Execute the third sub - model, where the third sub - model includes an output layer.

20. The home computing system according to claim 17, wherein The computer - executable instructions, when executed by the processor, further cause the home computing system to: Execute the first sub - model until a predetermined layer of the data analysis model is reached; and Send the processed data from the predetermined layer to the public computing cloud.

Images