Method, system and electronic device for managing sensor data

Abstract

Methods, systems, and electronic devices for managing sensor data include receiving, at a first device, encoded data from a second device separate from the first device, wherein the encoded data is generated using an artificial intelligence (AI) encoder model included in the second device based on sensor data collected by at least one sensor included in the second device; providing the encoded data to an AI reasoning model to obtain reasoning information; and executing a task based on the reasoning information, wherein the AI encoder model and the AI reasoning model are jointly trained based on the output of the AI teacher model.

Description

Technical Field

[0001] The present disclosure relates to managing sensor data, and more particularly to neural data compression of sensor data using knowledge distillation. Background Art

[0002] The International Telecommunication Union (ITU) Radiocommunication Sector (ITU-R) has three main use cases for the fifth generation (5G) mobile technology, namely enhanced mobile broadband (eMBB), ultra-reliable low latency communications (URLLC), and massive machine type communications (mMTC). Among these use cases, mMTC is designed to support the huge and increasing traffic generated by various applications such as Internet of Things (IoT) applications, body area networks, smart monitoring systems, smart homes, etc.

[0003] An important aspect supporting mMTC is network energy efficiency, which may refer to the ability of a node to minimize its energy consumption for radio access given a specific traffic capacity. A widely used technique to achieve improved network energy efficiency under load is data compression.

[0004] Many data compression techniques are intended for use in human type communications (HTC), where data is intended for human consumption at one end. Therefore, these data compression techniques may prioritize accurate reconstruction. However, in MTC and mMTC, the data generated is intended to be consumed by other machines, e.g., in order to implement a fully autonomous system. Therefore, mMTC implementations may benefit from data compression techniques that prioritize inferring accurate decisions.

[0005] For example, one or more sensors can be used to monitor certain phenomena of the environment. The sensors can be located in the edge nodes of the network, which can sense the environment through a series of observations, and then each sensor can send its sensor data to the central node, which fuses the sensor data and infers decisions based on the data. This setting is common in different fields and applications. Examples include intelligent abandoned object detection, in which one or more surveillance cameras monitor a space for abandoned objects that may contain hazardous materials; smoke detection, in which one or more surveillance cameras or other types of sensors monitor a space for smoke; or wearable sensor fusion, in which wearable devices are used to detect events associated with the user, such as health events or home automation events.

[0006] To extend the lifetime of nodes, save bandwidth, and reduce latency, each node can compress observations before transmitting to the central node. Because data in MTC and mMTC may be intended to be processed by machines, such as for reasoning about the environment, the decision of reasoning may be much more important than the reconstruction accuracy.

[0007] Therefore, data compression techniques are needed in which observations are compressed with the minimum possible loss in decision accuracy. Summary of the invention

[0008] Technical Solutions

[0009] Example embodiments address at least the above problems and / or disadvantages and other disadvantages not described above. Also, example embodiments are not required to overcome the disadvantages described above, and may not overcome any of the problems described above.

[0010] In an embodiment of the present disclosure, a method for managing sensor data includes: receiving encoded data at a first device from a second device separate from the first device, wherein the encoded data is generated based on sensor data collected by at least one sensor included in the second device and using an artificial intelligence (AI) encoder model included in the second device; providing the encoded data to an AI inference model to obtain inference information; and performing a task based on the inference information, wherein the AI ​​encoder model and the AI ​​inference model are jointly trained based on the output of an AI teacher model.

[0011] The size of the AI ​​inference model can be smaller than the size of the AI ​​teacher model.

[0012] The size of the encoded data may be smaller than the size of the sensor data.

[0013] The method may also include: obtaining, at the first device, a plurality of pieces of encoded data from a plurality of second devices separated from the first device, wherein the plurality of pieces of encoded data are generated using a plurality of AI encoder models included in the plurality of second devices; and combining the plurality of pieces of encoded data with the encoded data to generate aggregated data. The reasoning information may be generated by the AI ​​reasoning model based on the aggregated data, and the plurality of AI encoder models may be jointly trained with the AI ​​encoder model and the AI ​​reasoning model based on the output of the AI ​​teacher model.

[0014] The encoded data may be quantized by the AI ​​encoder model before being sent to the first device.

[0015] The second device may include a surveillance camera as at least one sensor, and the task may include detecting at least one of an object and an event observed by the surveillance camera.

[0016] The second device may include a wearable device, and the task may include detecting a health event associated with a user wearing the wearable device.

[0017] The second device may include an Internet of Things (IoT) device and may receive the encoded data using massive machine type communication (mMTC).

[0018] The AI ​​reasoning model may include a first neural network model, and the AI ​​teacher model may include at least one of a second neural network model, a support vector machine (SVM) model, and an ensemble model.

[0019] In an embodiment of the present disclosure, an electronic device for managing sensor data includes: at least one memory storing computer-readable instructions; and at least one processor configured to execute the computer-readable instructions to: receive encoded data at a first device from a second device separate from the first device, wherein the encoded data is generated based on sensor data collected by at least one sensor included in the second device and using an artificial intelligence (AI) encoder model included in the second device; provide the encoded data to an AI reasoning model to obtain reasoning information; and perform a task based on the reasoning information, wherein the AI ​​encoder model and the AI ​​reasoning model are jointly trained based on the output of an AI teacher model.

[0020] The size of the AI ​​reasoning model may be smaller than the size of the AI ​​teacher model. In an embodiment, the AI ​​reasoning model may be a student model of the AI ​​teacher model.

[0021] The size of the encoded data may be smaller than the size of the sensor data.

[0022] At least one processor may also be configured to execute computer-readable instructions to: obtain, at a first device, multiple pieces of encoded data from multiple second devices separate from the first device, wherein the multiple pieces of encoded data are generated using multiple AI encoder models included in the multiple second devices, and combine the multiple pieces of encoded data with the encoded data to generate aggregate data, wherein reasoning information is generated by the AI ​​reasoning model based on the aggregate data, and wherein the multiple AI encoder models are jointly trained based on the output of the AI ​​teacher model and the AI ​​encoder model and the AI ​​reasoning model.

[0023] The encoded data may be quantized by the AI ​​encoder model before being sent to the first device.

[0024] The second device may include a surveillance camera as at least one sensor, and the task may include detecting at least one of an object and an event observed by the surveillance camera.

[0025] The second device may include a wearable device, and the task may include detecting a health event associated with a user wearing the wearable device.

[0026] The second device may include an Internet of Things (IoT) device and may receive the encoded data using massive machine type communication (mMTC).

[0027] The AI ​​reasoning model may include a first neural network model, and the AI ​​teacher model may include at least one of a second neural network model, a support vector machine (SVM) model, and an integrated model.

[0028] In an embodiment of the present disclosure, a non-transitory computer-readable storage medium stores instructions that, when executed by at least one processor of an electronic device for managing sensor data, cause the at least one processor to: receive encoded data at a first device from a second device separate from the first device, wherein the encoded data is generated based on sensor data collected by at least one sensor included in the second device and using an artificial intelligence (AI) encoder model included in the second device; provide the encoded data to an AI reasoning model to obtain reasoning information; and perform a task based on the reasoning information, wherein the AI ​​encoder model and the AI ​​reasoning model are jointly trained based on the output of an AI teacher model.

[0029] The size of the AI ​​inference model is smaller than the size of the AI ​​teacher model, and wherein the size of the encoded data is smaller than the size of the sensor data. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The above and other aspects, features and aspects of the embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings, in which:

[0031] Figure 1 is a diagram showing a general overview of a system for managing sensor data according to an embodiment;

[0032] Figure 2 is a diagram showing an example structure of an autoencoder according to an embodiment;

[0033] Figure 3 is a diagram illustrating a system for training a student model according to an embodiment;

[0034] Figure 4 is a diagram showing an example of data quantization according to an embodiment;

[0035] Figure 5 is a diagram illustrating a system for jointly training a distillation encoder and an inference model according to an embodiment;

[0036] Fig. 6Ais a flow chart illustrating the overall process of training a teacher model, a distillation encoder, and an inference network to manage sensor data according to an embodiment;

[0037] Figure 6B is a flow chart illustrating a process of jointly training a distillation encoder and an inference model according to an embodiment;

[0038] Figure 7 is a flow chart of a process for using an inference model to perform a task based on data received from a distillation encoder.

[0039] FIG. 8A to FIG. 8C is a diagram showing an example of a use application of a distillation encoder and an inference model according to an embodiment;

[0040] 9A to 9C shows an example dataset for testing a distillation encoder and an inference model according to an embodiment;

[0041] Fig.9D shows example results of performing a classification task using a distillation encoder and an inference model according to an embodiment; and

[0042] Fig.10 is a block diagram of an electronic device according to an embodiment. DETAILED DESCRIPTION

[0043] Example embodiments are described in more detail below with reference to the accompanying drawings.

[0044] In the following description, the same reference numerals are used for the same elements even in different drawings. Matters defined in the specification, such as detailed construction and elements, are provided to assist in a comprehensive understanding of the exemplary embodiments. However, it is apparent that the exemplary embodiments can be practiced without those specifically defined matters. In addition, well-known functions or structures are not described in detail because they would obscure the description with unnecessary detail.

[0045] Expressions such as "at least one of..." when preceding a list of elements modify the entire list of elements rather than modifying the individual elements in the list. For example, the expression "at least one of a, b, and c" should be understood to include only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or any variation of the foregoing examples.

[0046] Although terms such as "first", "second", etc. may be used to describe various elements, these elements must not be limited to the above terms. The above terms may be used only to distinguish one element from another.

[0047] The term "component" is intended to be broadly interpreted as hardware, firmware, or a combination of hardware and software.

[0048] It is apparent that the systems and / or methods described herein can be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit the implementation. Therefore, the operation and behavior of the systems and / or methods are described herein without reference to specific software code, and it should be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0049] Even though particular combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features may be combined in ways not specifically recited in the claims and / or disclosed in the specification. Although each dependent claim listed below may be directly dependent on only one claim, the disclosure of possible implementations includes the combination of each dependent claim with every other claim in the claim set.

[0050] Unless explicitly described as such, elements, actions, or instructions used herein should not be interpreted as critical or essential. In addition, as used herein, the articles "a" and "an" are intended to include one or more items and can be used interchangeably with "one or more." In addition, as used herein, the term "set" is intended to include one or more items (e.g., related items, unrelated items, combinations of related and unrelated items, etc.) and can be used interchangeably with "one or more." Figure 1 In the case of multiple items, the term "a" or similar language is used. In addition, as used herein, the terms "has," "have," "having," and the like are intended to be open-ended terms. In addition, unless expressly stated otherwise, the phrase "based on" is intended to mean "based, at least in part, on."

[0051] Figure 1 is a diagram showing a general overview of a system for managing sensor data according to an embodiment.

[0052] Figure 1 is a diagram showing a general overview of a system 100 for predicting future loads according to an embodiment. The system 100 may be a distributed reasoning network that may be used to perform tasks based on sensor data. For example, the system 100 may be used to make decisions or generate alerts or notifications based on events or objects detected using sensor data.

[0053] System 100 may include sensor nodes 110 and a fusion center 120. Figure 1 Only one sensor node 110 is shown, but the embodiment is not limited thereto, and the system 100 may include multiple sensor nodes 110. The sensor node 110 may send sensor data to the fusion center 120, and the fusion center 120 may perform tasks such as making decisions or generating alarms or notifications based on the objects, activities, and / or events detected by the sensor node 110.

[0054] The sensor node 110 may be any device capable of obtaining sensor data, such as a surveillance camera, a wearable device, an edge device, etc. In an embodiment, the sensor node 110 may be, for example, an Internet of Things (IoT) device. The sensor node 110 may include a sensor 111, a distillation encoder 112, and a transmitter 113. The sensor 111 may be any type of sensor, such as an image sensor, a microphone, an accelerometer, a gyroscope, a magnetometer, a position sensor such as a global positioning system (GPS) sensor, a heart rate sensor, a pedometer, a motion sensor, a pressure sensor, etc. For example, based on the sensor node 110 being a surveillance camera, the sensor 111 may include an image sensor and / or a microphone. As another example, based on the sensor node 110 being a wearable device, the sensor 111 may include at least one of a GPS sensor, an accelerometer, a heart rate sensor, and / or a pedometer. As yet another example, based on the sensor node 110 being a security sensor, the sensor 111 may include a motion sensor and / or a pressure sensor. The distillation encoder 112 may receive raw sensor data from the sensor 111 and may encode the raw data to generate encoded sensor data, for example, as a codeword. The transmitter 113 may be a wired or wireless transmitter that may transmit the encoded sensor data to the fusion center 120. For example, the transmitter 113 may transmit the encoded sensor data using massive machine type communication (mMTC).

[0055] The fusion center 120 may be a central node of a distributed reasoning network included in the system 100. The fusion center 120 may include a receiver 121 and a reasoning model 122. The receiver 121 may be a wired or wireless receiver that may receive encoded sensor data from the sensor node 110. The reasoning model 122 may process the encoded sensor data to generate reasoning information that may be used to perform tasks such as making decisions or generating alarms or notifications. For example, based on the sensor node 110 being a surveillance camera, the reasoning model 122 may generate reasoning information indicating objects, activities, and / or events detected by the surveillance camera, such as abandoned objects, and the fusion center 120 may generate notifications or alarms based on the reasoning information. As another example, based on the sensor node 110 being a wearable device, the reasoning model 122 may generate reasoning information about the environment of a user wearing the wearable device, such as information about the user's location or a health event of the user, and the fusion center may generate a notification or alarm based on the reasoning information, or may perform tasks such as home automation tasks based on the reasoning information. As yet another example, based on the sensor node 110 being a security sensor, the inference model 122 may generate inference information about a security status of a facility or a home, and may trigger an alarm based on the inference information.

[0056] In an embodiment, the distillation encoder 112 and the reasoning model 122 may be an artificial intelligence (AI) model, such as a neural network model or a machine learning model. In an embodiment, the distillation encoder 112 and the reasoning model 122 may be jointly trained using a knowledge distillation process, for example, based on the output of a teacher model. Fig. 6A and Figure 6B An example of a training process for training the distillation encoder 112 and the inference model 122 is described.

[0057] Figure 2 is a diagram showing an example structure of an autoencoder according to an embodiment.

[0058] like Figure 2 As shown, the autoencoder 200 can be a neural network architecture including an encoder model 201 and a decoder model 202. The encoder model 201 can nonlinearly transform the input data X into a low-dimensional codeword C. The space of the encoded codewords C can be called a latent space, which can be lower dimensional than the input space. Then, the decoder model 202 can decode the codewords C to reconstruct the original input data X, for example, by generating reconstructed data Autoencoders can be used to compress and decompress data such as images, videos, wireless channels, biological signals, or brain-computer interaction data.

[0059] Figure 3 is a diagram illustrating a system for training a student model using knowledge distillation according to an embodiment.

[0060] like Figure 3 As shown, the system 300 may include a training data set 301 and a student model 303 (which may also be referred to as an AI reasoning model in this article) and a teacher model 302 (which may also be referred to as an AI teacher model in this article). In an embodiment, the data set 301 may include original data X and ground truth Y, which may be used to train the teacher model 302 and may be further used to calculate the distillation loss, which may be used to train the student model 303.

[0061] Knowledge distillation is a model training technique that can be used to distill the knowledge of a large and powerful model (e.g., teacher model 302) into a smaller and less powerful model (e.g., student model 303). Using knowledge distillation, student model 303 can be trained to mimic the behavior of teacher model 302. Although teacher model 302 consumes high computing and power resources, which hinders its deployment to resource-constrained edge devices such as sensor nodes 110, student model 303 can be lighter and easier to deploy. Therefore, knowledge distillation can be used to compress cumbersome models to obtain smaller models that can be deployed to resource-limited edge devices without significantly losing model performance. Knowledge distillation can be used in many fields, such as object detection, semantic segmentation, etc.

[0062] like Figure 3 As shown, the student model can be trained using the distillation loss, which uses the ground truth data Y and the output Y of the teacher model generated based on the original input data X. t And the output Y of the student model generated based on the original input data X s For example, the objective function of the student model 303 can be formulated as a convex combination of task loss and knowledge distillation loss. In an embodiment, the distillation loss can be expressed according to the following equation 1:

[0063]

[0064] In equation 1, y i represents the ground truth value of the i-th data point in the dataset 301, represents the output of the student model 303 for the ith data point, represents the output of the teacher model 302 for the i-th data point, represents the weight term that controls the contribution of each term in the final loss, CE represents the cross entropy loss, and KL represents the Kullback-Leibler divergence. In an embodiment, the ground truth values ​​in the dataset 301 can be referred to as hard labels or one-hot encoded labels, and the output of the teacher model 302 can be referred to as soft labels. In one embodiment,

[0065] The cross entropy loss CE may be an example of a task loss for a classification task. In an embodiment, the cross entropy loss CE may be determined according to the following equation 2, where N represents the number of categories:

[0066]

[0067] Therefore, the cross entropy loss CE of the i-th data point can be expressed according to the following equation 3:

[0068]

[0069] The Kullback-Leibler divergence KL may be an example of a knowledge distillation loss. In an embodiment, the Kullback-Leibler divergence KL may be determined according to the following equation 4:

[0070]

[0071] In equation 4, Y s represents the distribution of predictions of the student model 303, and Y t represents the distribution of predictions of the teacher model 302.

[0072] Therefore, the Kullback-Leibler divergence KL of the i-th data point can be expressed according to the following equation 5:

[0073]

[0074] In an embodiment, Y t It can be called a soft label and can be the output of a Softmax layer that can be included in the teacher model 302. For example, the Softmax layer can convert the logit (multivariate logistic) gi generated by the teacher model 302 for each category into the probability qi of each category according to the following equation 6:

[0075]

[0076] In Equation 6 above, T may represent a temperature that controls the softness of the generated distribution. Typically, T may be set to 1, and higher values ​​of T may generate softer probability distributions over classes. These soft labels may contain information about the correlation between classes that may not be included in the ground truth hard labels (e.g., one-hot encoded labels).

[0077] Figure 4 is a diagram illustrating an example of data quantization according to an embodiment.

[0078] Quantization may refer to the process of mapping input values ​​in a large set (e.g., a continuous set that may be infinite) to output values ​​in a countable smaller set (e.g., a set with a countable or finite number of elements). Thus, a quantizer may refer to a specific input / output function mapping that may be used to implement or perform a quantization process on input data.

[0079] Quantized data can be sent between machines more efficiently than unquantized data. Quantization can also be used in deep learning to help speed up inference and reduce memory and power consumption on embedded devices.

[0080] Some examples of quantization include rounding and truncation. For example, quantized random projections of the data can be sent to a cloud server. Random projections can provide rate-efficient representation and time-efficient computation with negligible performance loss for downstream tasks.

[0081] according to Figure 4 In the example shown, the real values ​​of a continuous signal 401 are converted into words 402 having a length of 2 bits.

[0082] Figure 5 is a diagram illustrating a system for jointly training a distillation encoder and an inference model according to an embodiment.

[0083] like Figure 5 As shown, the system 500 may include a dataset 501 and a teacher model 502, a distillation encoder 112, and an inference model 122. In an embodiment, the dataset 501 may include original data X and ground truth Y, which may be used to train the teacher model 502 and may also be used to calculate the distillation loss, which may be used to train the distillation encoder 112 and the inference model 122.

[0084] In an embodiment, system 500 may be similar to system 300, except that instead of student model 303, system 500 may include distillation encoder 112 and inference model 122. In an embodiment, distillation encoder 112 and inference model 122 may be jointly trained using a distillation loss, which may be calculated using dataset 501 and the output of teacher model 502. For example, in an embodiment, the distillation loss may be calculated using Equation 1, where y i can represent the ground truth value of the i-th data point in the dataset 501, may represent the output of the inference model 122 for the i-th data point (which may be generated based on the output of the distillation encoder 112 based on the i-th data point), and The output of the teacher model 502 for the i-th data point may be represented.

[0085] In an embodiment, joint training of the distillation encoder 112 with the inference model 122 may allow the system 100 to minimize mispredictions at the fusion center 120. For example, the inference model 122 may inherit the knowledge of the teacher model 502, which is trained to infer decisions based on raw data without compression. In this case, the teacher model 502 may be a knowledgeable model trained to infer decisions given complete knowledge of the data, for example, by predicting a conditional probability P(Y|X), where Y represents a target label and X represents the raw input data. The inference model 122 may be trained to infer decisions given only partial information, for example, by predicting a conditional probability P(Y|C), where C is compressed data generated by the distillation encoder 112 based on the raw input data X.

[0086] When the knowledge captured by the teacher model 502 is distilled to the inference model 122, the distillation encoder 112 can learn to encode the features most relevant to the inference model 122 to maximize its accuracy, and thus can compress the original input data with the minimum possible loss in prediction accuracy. Therefore, the conditional distribution P(Y|C) can be close to the conditional distribution P(Y|X).

[0087] Therefore, similar to the above Figure 3In contrast to the knowledge distillation techniques discussed, the input to the teacher model 502 can be much larger than the input to the inference model 122, which can operate based on the compressed input provided by the distillation encoder 112. Therefore, by minimizing the KL divergence between the probability distributions generated by the teacher model 502 and the inference model 122, the inference model 122 can generalize as well as the teacher model 502, even though the inference model may be smaller and / or less resource intensive than the teacher model 502, and the inference model 122 can operate on input that may be less resource intensive to generate, send, and / or store than the original input data used by the teacher model 502.

[0088] In an embodiment, the teacher model 502 may be any type of model, such as any type of AI model and / or ML model. For example, the teacher model may be or may include a neural network (NN) of any architecture, a support vector machine (SVM), an ensemble model, a random forest (RF) model, or any other type of model that may be used to generate soft labels.

[0089] In an embodiment, the teacher model 502 may be trained on a compressed data set instead. For example, an encoder may be used to generate compressed data X corresponding to the original input data X. c , to generate the compressed data X c Therefore, the teacher model 502 can be trained to predict the conditional distribution P(Y|X c ), rather than being trained to predict the conditional distribution P(Y|X). In an embodiment, the distillation encoder 112 may be a quantized encoder that may generate quantized outputs that are provided to the inference model 122. For example, the distillation encoder 112 may be configured to generate a set of discrete finite values ​​(e.g., Z∈{0,1} m ) rather than consisting of a set of continuous infinite values ​​(e.g., C∈R n ). Therefore, the inference model 122 can be trained to predict the conditional distribution P(Y|Z) instead of being trained to predict the conditional distribution P(Y|C).

[0090] In an embodiment, the teacher model 502 may be pre-trained, and the output of the teacher model 502 may be obtained before training the distillation encoder 112 and the inference model 122. For example, in an embodiment, the conditional distribution P(Y|X) (or the conditional distribution P(Y|X c )) can be obtained in advance and can be included in the data set 501 or stored in addition to the data set 501. Therefore, it is possible to obtain the conditional distribution P(Y|X) based on the data set 501 and the pre-stored conditional distribution P(Y|X) (or the conditional distribution P(Y|X) c )) to train the distillation encoder 112 and the inference model 122.

[0091] Therefore, compared to other data compression techniques that prioritize high reconstruction accuracy, the distillation encoder 112 can achieve high compression ratios while maintaining the accuracy of the inference decisions made by the inference model 122. Therefore, the distillation encoder 112 can be used to perform data compression and / or quantization in mMTC communications, and embodiments can utilize knowledge distillation to efficiently compress and / or quantize sensor data.

[0092] although Figure 5 Only one distillation encoder 112 is shown, but the embodiment is not limited thereto, and the system 500 can be used to train the inference model 122 using the combined or aggregated outputs of multiple distillation encoders 112. For example, each of the multiple distillation encoders 112 can receive some or all of the original input data X and can generate a corresponding codeword C (or quantized codeword Z).

[0093] Fig. 6A is a flowchart illustrating the overall process of training a teacher model, a distillation encoder, and an inference network to manage sensor data according to an embodiment.

[0094] In operation 611, process 610 may include obtaining a training dataset. In an embodiment, the training dataset may correspond to dataset 501, which may include original input data X and ground truth values ​​Y.

[0095] In operation 612, process 610 may include training the teacher model to minimize the cross entropy loss with respect to the training data set. For example, as described above, in an embodiment, the teacher model 502 may be trained using the original input data X. In an embodiment, the teacher model 502 may be trained based on a compressed version of the input data X.

[0096] In operation 613, process 610 may include generating soft labels using the teacher model. For example, after the teacher model 502 converges, the trained teacher model 502 may be used to generate soft labels Y based on the data set 501. t .

[0097] In operation 614 , process 610 may include concatenating the soft labels generated by the teacher model with the ground truth labels.

[0098] At operation 615 , process 610 may include jointly training the distillation encoder and the inference model using knowledge distillation to minimize the distillation loss.

[0099] In an embodiment, the distillation encoder may correspond to the distillation encoder 112, and the inference model may correspond to the inference model 122. The distillation loss may be calculated according to the above equation 1. As described above, in an embodiment, the distillation encoder 112 may be or may include a quantization encoder, and the inference model 122 may be trained using a quantized version of the input data X.

[0100] Figure 6B is a flow chart illustrating a process of jointly training a distillation encoder and an inference model according to an embodiment. In an embodiment, Fig. 6A One or more of operations 614 and 615 may include Figure 6B One or more of operations 621-626 in.

[0101] In operation 621, process 620 may include concatenating the soft label with the ground truth hard label. In an embodiment, operation 621 may correspond to Fig. 6A Operation 614 in.

[0102] At operation 622, process 620 may include applying the weight term Set equal to 1.

[0103] In operation 623, process 620 may include training the distillation encoder and the inference model e epochs to minimize the distillation loss. In one embodiment, the distillation loss may be calculated according to Equation 1 above.

[0104] At operation 624, process 620 may include determining Is it greater than 0.5?

[0105] Based on determination Greater than 0.5 (eg, yes at operation 624 ), process 620 may return to operation 622 .

[0106] Based on determination If the value is not greater than 0.5 (e.g., no at operation 624), process 620 may proceed to operation 626, which may include training the distillation encoder and the inference model for r additional rounds to minimize the distillation loss. After operation 626, the training of the distillation encoder and the inference model may be completed.

[0107] In an embodiment, one or more of the processes 610 and 620 may be referred to as a training phase of the distillation encoder 112 and the inference model 122. During the training phase, the knowledge of the teacher model 502 may be distilled to the inference model 122, which operates based on a compressed and / or quantized version of the original data that may be provided by the distillation encoder 112.

[0108] After training, the distillation encoder 112 and the inference model 122 can be deployed to help manage sensor data, which can be referred to as a deployment phase. For example, in the deployment phase, the distillation encoder 112 can be deployed at the sensor node 110, and the inference model 122 can be deployed at the fusion center 120. Figure 7 and FIG. 8A to FIG. 8C An example that discusses the deployment phase in more detail.

[0109] Figure 7 is a flow chart of a process for using an inference model to perform a task based on data received from a distillation encoder.

[0110] In operation 701 , process 700 may include receiving, at a first device, encoded data from a second device.

[0111] In an embodiment, the first device may refer to the fusion center 120, and the second device may refer to the sensor node 110. The encoded data may be generated by an encoder model, which may correspond to the distillation encoder 112, and may be jointly trained with the inference model, which may correspond to the inference model 122. The encoded data may refer to at least one of a codeword C and a quantized codeword Z, which may be generated using the encoder model based on sensor data collected by at least one sensor included in the second device. The joint training may refer to at least one of the processes 610 and 620 discussed above.

[0112] In operation 702, process 700 may include providing the encoded data to an inference model to obtain inference information. In an embodiment, the inference information may refer to an output of the inference model 122. In an embodiment, the inference information may indicate a decision or result obtained by the inference model 122, which may be used to trigger or perform a task.

[0113] In operation 703, process 700 may include performing a task based on the inference information. In an embodiment, the task may be or may include making a decision or generating an alert or notification based on an event, activity, or object detected using sensor data.

[0114] In an embodiment, the size of the inference model may be smaller than the size of the teacher model.

[0115] In an embodiment, the size of the encoded data may be smaller than the size of the sensor data.

[0116] In an embodiment, process 700 may also include obtaining multiple pieces of encoded data from multiple second devices separated from the first device at the first device. Multiple encoder models included in the multiple second devices may be used to generate the multiple pieces of encoded data. Multiple encoder models may be jointly trained with the encoder model and the inference model based on the output of the teacher model.

[0117] In an embodiment, process 700 may also include combining the plurality of pieces of encoded data with the encoded data to generate aggregated data. Inference information may be generated by an AI inference model based on the aggregated data.

[0118] In an embodiment, the encoded data may be quantized by an encoder model before being sent to the first device.

[0119] In an embodiment, the second device may include a surveillance camera as the at least one sensor, and the task may include detecting at least one of an object and an event observed by the surveillance camera.

[0120] In an embodiment, the second device may include a wearable device, and the task includes detecting a health event associated with a user wearing the wearable device.

[0121] In an embodiment, the second device may include an IoT device and may receive the encoded data using mMTC.

[0122] Reference below FIG. 8A to FIG. 8C Examples of use applications in which the distillation encoder 112 and the inference model 122 may be deployed are discussed in more detail.

[0123] Fig. 8A The use application of deploying multiple distillation encoders and inference models in a supervision network is shown.

[0124] like Fig. 8A As shown, the use application 810 may include multiple monitoring cameras 811 and a fusion center 812. Each monitoring camera 811 may correspond to a sensor node 110, and may include a distillation encoder 112 and a sensor 111, and the sensor 111 may be, for example, at least one of an image sensor, a microphone, a mobile sensor, etc. The fusion center 812 may correspond to the fusion center 120, and may include an inference model 122 jointly trained with the distillation encoder 112 included in the monitoring camera 811.

[0125] The surveillance camera 811 may be used to monitor the surveillance area 813, and may provide encoded data to the fusion center 812. The encoded data may be generated by the distillation encoder 112 based on the raw sensor data captured by the sensor 111. The reasoning model 122 may generate reasoning information based on the encoded data, and the reasoning information may be used to perform tasks associated with the surveillance camera 811. For example, the reasoning information may be information indicating that an activity, event, or object exists or has occurred in the surveillance area 813, and the task may include detecting the activity, event, or object, generating an alert, alarm, or notification corresponding to the activity, event, or object, and performing at least one of some other actions corresponding to the activity, event, or object.

[0126] In an embodiment, the size of the encoded data may be smaller than the original sensor data. Therefore, the bandwidth, transmission power and / or storage space used by the surveillance camera 811 may be reduced.

[0127] In an embodiment, using surveillance cameras 811 and fusion center 812 to automate the task of monitoring critical spaces, such as monitoring area 813, can save human resources and costs. In addition, automated monitoring can be more accurate than human-based monitoring. For example, automated monitoring provided by surveillance cameras 811 and fusion center 812 can be used to detect abandoned objects, such as abandoned bags in an airport, or to detect events such as fires or unauthorized personnel entering monitoring area 813.

[0128] Figure 8B A use application is shown where multiple distilled encoders and inference models are deployed in a body area network.

[0129] like Figure 8B As shown, the use application 820 may include multiple wearable devices 821 and a fusion center 822. Each wearable device 821 may correspond to a sensor node 110, and may include a distillation encoder 112 and a sensor 111, and the sensor 111 may be, for example, a heart rate sensor, a position sensor, an accelerometer, etc. The fusion center 822 may correspond to the fusion center 120, and may include an inference model 122 jointly trained with the distillation encoder 112 included in the wearable device 821.

[0130] In an embodiment, the size of the encoded data may be smaller than the original sensor data. Therefore, the bandwidth, transmission power and / or storage space used by the wearable device 821 may be reduced.

[0131] The wearable device 821 may be worn by a user 823 and may provide encoded data to a fusion center 822. The encoded data may be generated by a distilled encoder 112 based on raw sensor data captured by the sensor 111. The inference model 122 may generate inference information based on the encoded data, which may be used to perform a task associated with the wearable device 821. For example, the inference information may be information indicating that an activity or event associated with the user 823 has occurred, and the task may include detecting the activity or event, generating an alarm, alert, or notification corresponding to the activity or event, and performing at least one of some other action corresponding to the activity, event, or object. For example, the use application 820 may be used to generate a health alert related to the user 823, or to perform a home automation task corresponding to the user 823.

[0132] Figure 8C A use application involving multiple quantized distillation encoders is shown.

[0133] like Figure 8C As shown, the use application 830 may include a plurality of wearable devices 831 and a fusion center 832. The use application 830 may be similar to the use application 820, except that the wearable device 831 may transmit quantized encoded data. For example, based on the original input data X=[x 1 , x 2 , ..., x n ]∈R n , the distillation encoder 112 included in the wearable device 831 may generate quantized encoded data X=[x 1 , x 2 , ..., x m ]∈{0,1} m In an embodiment, the quantized encoded data may be sent to the fusion center 832 in real time, or may be stored locally to be sent or retrieved at a later time.

[0134] In an embodiment, the size of the quantized encoded data may be smaller than the original sensor data and the encoded data using the application 820. Therefore, the bandwidth, transmission power, and / or storage space used by the wearable device 831 may be further reduced relative to the wearable device 821.

[0135] Although usage application 830 is presented as a variation of usage application 820 , embodiments are not limited thereto, and usage application 830 may be similarly modified to use a quantized version of distillation encoder 112 .

[0136] 9A to 9C An example dataset for testing a distillation encoder and an inference model according to an embodiment is shown. Specifically, Fig.9A shows an example of data from the Modified National Institute of Standards and Technology (MNIST) database, Fig. 9B shows an example of data from the Fashion-MNIST database, and Fig. 9C Data from the Canadian Institute for Advanced Research (CIFAR)-10 database is shown. Table 1 below shows the size, dimension, and number of categories in each dataset, as well as the size of the test set used to test the distillation encoder and inference model according to the embodiment.

[0137] Table 1

[0138] Dataset MNIST Fashion-MNIST CIFAR10 Training set size 50000 5000 50000 Test set size 1000 1000 1000 Dimensions 28x28 28x28 32x32x3 Number of categories 10 10 10

[0139] For the MNIST and Fashion-MNIST datasets, a distillation encoder and inference model according to an embodiment are trained using a teacher model comprising two hidden layers activated with rectified linear units (ReLU) of 200 and 100 nodes, each followed by a dropout layer with a rate of 0.2. For the CIFAR10 dataset, a distillation encoder and inference model according to an embodiment are trained using a teacher model with a convolutional neural network-based architecture comprising three visual geometry group (VGG) blocks, each with 32, 64, and 128 filters. For the CIFAR10 teacher model, the layers are initialized using He initialization, and ReLU activation is used for each layer.

[0140] For each dataset, an inference model including a single-layer classifier and a distillation encoder including a single-layer classifier were used. Stochastic gradient descent (SGD) was performed with a learning rate = 0.001, and a 0.9 momentum was used to optimize the cross entropy loss function. The output layer of all models was a Softmax layer, where the number of nodes was equal to the number of categories.

[0141] Table 2 shows an example of accuracy gain obtained by using a distillation encoder and an inference model according to an embodiment, and Table 3 shows an example of accuracy gain obtained by using a quantized distillation encoder and an inference model according to an embodiment. It can be seen from Tables 2 and 3 that using a distillation encoder and an inference model according to an embodiment can result in considerable accuracy gain.

[0142] Table 2

[0143]

[0144] Table 3

[0145]

[0146] Fig.9D Example results of using a distillation encoder and an inference model to perform a classification task according to an embodiment are shown. In addition, Fig.9D Shown are the results obtained after distillation from two non-neural network teacher models (specifically, the SVM teacher model and the RF teacher model).

[0147] Above about Figure 9A-9D The results presented in Tables 1, 2, and 3 are presented only as examples, and the embodiments are not limited thereto.

[0148] Fig.10 is a block diagram of an electronic device according to an embodiment.

[0149] Fig.10 This is for illustration only, and other embodiments of the electronic device 1000 may be used without departing from the scope of the present disclosure. For example, the electronic device 1000 may correspond to at least one of the sensor node 110 and the fusion center 120.

[0150] The electronic device 1000 includes a bus 1010 , a processor 1020 , a memory 1030 , an interface 1040 , and a display 1050 .

[0151] The bus 1010 includes a circuit for connecting the components 1020 to 1050 to each other. The bus 1010 serves as a communication system for transmitting data between the components 1020 to 1050 or between electronic devices.

[0152] The processor 1020 includes one or more of a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), many integrated cores (MIC), a field programmable gate array (FPGA), or a digital signal processor (DSP). The processor 1020 is capable of performing control of any one or any combination of other components of the electronic device 1000, and / or performing communication-related operations or data processing. For example, the processor 1020 may perform Fig. 6A The process 610 shown in Figure 6B The process 620 shown in and Figure 7 The operations of process 700 are shown in FIG. 1 . Processor 1020 executes one or more programs stored in memory 1030 .

[0153] The memory 1030 may include volatile and / or nonvolatile memory. The memory 1030 stores information related to at least one other component of the electronic device 1000 and used to drive and control the electronic device 1000, such as one or more of commands, data, programs (one or more instructions), applications 1034, etc. For example, the commands and / or data may constitute an operating system (OS) 1032. The information stored in the memory 1030 may be executed by the processor 1020.

[0154] Applications 1034 include the embodiments discussed above. These functions may be performed by a single application or multiple applications, each of which performs one or more of these functions. For example, application 1034 may include a program for executing Fig. 6A The process 610 shown, Figure 6B The process 620 and Figure 7 An artificial intelligence (AI) model of the operation of the illustrated process 700. Specifically, the application 1034 may include at least one of a distillation encoder and an inference model according to an embodiment of the present disclosure.

[0155] The display 1050 includes, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a quantum dot light emitting diode (QLED) display, a microelectromechanical system (MEMS) display, or an electronic paper display.

[0156] Interface 1040 includes an input / output (I / O) interface 1042, a communication interface 1044, and / or one or more sensors 1046. I / O interface 1042 serves as an interface that can transmit commands and / or data between a user and / or other external devices and other components of electronic device 1000, for example.

[0157] The communication interface 1044 may include a transceiver to enable communication between the electronic device 1000 and other external devices (e.g., a sensor node or a fusion center) via a wired connection, a wireless connection, or a combination of a wired and wireless connection. The communication interface 1044 may allow the electronic device 1000 to receive information from another device and / or provide information to another device. For example, the communication interface 1044 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, etc.

[0158] The transceiver of the communication interface 1044 may include a radio frequency (RF) circuit and a baseband circuit.

[0159] The baseband circuit can send and receive signals through a wireless channel, and can perform frequency band conversion and amplification on the signal. The RF circuit can up-convert the baseband signal provided from the baseband circuit to an RF band signal, and then send the converted signal through the antenna, and down-convert the RF band signal received through the antenna to a baseband signal. For example, the RF circuit may include a transmit filter, a receive filter, an amplifier, a mixer, an oscillator, a digital-to-analog converter (DAC), and an analog-to-digital converter (ADC).

[0160] The transceiver can be connected to one or more antennas. The RF circuitry of the transceiver can include multiple RF chains and can perform beamforming. For beamforming, the RF circuitry can control the phase and magnitude of each signal sent and received through multiple antennas or antenna elements. The RF circuitry can perform downlink multiple-input and multiple-output (MIMO) operations by sending one or more layers.

[0161] The baseband circuit can perform conversion between baseband signals and bit streams according to the physical layer standard of the radio access technology. For example, when sending data, the baseband circuit generates complex symbols by encoding and modulating the transmission bit stream. When receiving data, the baseband circuit reconstructs the received bit stream by demodulating and decoding the baseband signal provided by the RF circuit.

[0162] The sensor 1046 of the interface 1040 can measure physical quantities or detect the activation state of the electronic device 1000, and convert the measured or detected information into an electrical signal. For example, the sensor 1046 may include one or more cameras or other imaging sensors for capturing scene images. The sensor 1046 may also include any one or any combination of a microphone, a keyboard, a mouse, and one or more buttons for touch input. The sensor 1046 may also include an inertial measurement unit. In addition, the sensor 1046 may include a control circuit for controlling at least one sensor included herein. Any of these sensors 1046 may be located within the electronic device 1000 or coupled to the electronic device 1000.

[0163] The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the implementations to the precise forms disclosed. Modifications and variations are possible in light of the above disclosure or may be acquired from practice of the implementations.

[0164] As used herein, the term "component" is intended to be broadly interpreted as hardware, firmware, or a combination of hardware and software.

[0165] It is apparent that the systems and / or methods described herein can be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit the implementation. Therefore, the operation and behavior of the systems and / or methods are described herein without reference to specific software code, and it should be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0166] Even though particular combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features may be combined in ways not specifically recited in the claims and / or disclosed in the specification. Although each dependent claim listed below may be directly dependent on only one claim, the disclosure of possible implementations includes the combination of each dependent claim with every other claim in the claim set.

[0167] The above-described embodiments of the present disclosure may be written as computer-executable programs or instructions that may be stored in a medium.

[0168] The medium can store computer executable programs or instructions continuously, or temporarily store computer executable programs or instructions for execution or download. In addition, the medium can be any of various recording media or storage media in which single or multiple pieces of hardware are combined, and the medium is not limited to the medium directly connected to the electronic device 1000, but can be distributed on the network. Examples of media include magnetic media (such as hard disks, floppy disks, and tapes), optical recording media (such as CD-ROMs and DVDs), magneto-optical media (such as optical magnetic floppy disks), and ROM, RAM, and flash memory, which are configured to store program instructions. Other examples of media include recording media and storage media managed by application stores that distribute applications or by websites, servers, etc. that supply or distribute various other types of software.

[0169] The above methods and processes may be provided in the form of downloadable software. A computer program product may include a product in the form of a software program electronically distributed by a manufacturer or an electronic market (e.g., a downloadable application). For electronic distribution, at least a portion of the software program may be stored in a storage medium or may be temporarily generated. In this case, the storage medium may be a storage medium of a server or electronic device 1000.

[0170] The model related to the above-mentioned neural network can be implemented via a software module. When the model is implemented via a software module (eg, a program module including instructions), the model can be stored in a computer-readable recording medium.

[0171] In addition, the model can be a part of the above-mentioned electronic device 1000 integrated in the form of a hardware chip. For example, the model can be manufactured in the form of a dedicated hardware chip for artificial intelligence, or can be manufactured as a part of an existing general-purpose processor (e.g., a CPU or an application processor) or a graphics-specific processor (e.g., a GPU).

[0172] In addition, the model can be provided in the form of downloadable software. The computer program product can include a product in the form of a software program that is electronically distributed by a manufacturer or an electronic market (e.g., a downloadable application). For electronic distribution, at least a portion of the software program can be stored in a storage medium or can be temporarily generated. In this case, the storage medium can be a server of the manufacturer or electronic market, or a storage medium of a relay server.

[0173] In the present disclosure, a model may be composed of a plurality of neural network layers. Each of the plurality of neural network layers has a plurality of weight values, and a neural network operation is performed by an operation between the operation result of the previous layer and the plurality of weight values. The plurality of weights owned by the plurality of neural network layers may be optimized by the learning result of the model. For example, the plurality of weights may be updated to reduce or minimize the loss value or cost value obtained from the model during the learning process. For example, the model may include a convolutional neural network (CNN), a deep neural network (DNN), a recursive neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), and a bidirectional recursive deep neural network (BRDNN) or a deep Q network, etc., but is not limited to the above examples.

[0174] Although embodiments of the present disclosure have been described with reference to the accompanying drawings, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope defined by the following claims.

Claims

1. A method 700 for managing sensor data, the method 700 comprising: Receiving (701) encoded data at a first device from a second device separate from the first device, wherein the encoded data is generated based on sensor data collected by at least one sensor included in the second device using an artificial intelligence (AI) encoder model included in the second device; providing (702) the encoded data to an AI reasoning model to obtain reasoning information; and performing (703) a task based on the inference information, Wherein, the AI ​​encoder model and the AI ​​reasoning model are jointly trained based on the output of the AI ​​teacher model.

2. The method 700 according to claim 1, wherein: The size of the AI ​​reasoning model is smaller than the size of the AI ​​teacher model.

3. The method 700 according to any one of claims 1 and 2, further comprising: obtaining, at the first device, a plurality of pieces of encoded data from a plurality of second devices separate from the first device, wherein the plurality of pieces of encoded data are generated using a plurality of AI encoder models included in the plurality of second devices; and combining the plurality of pieces of encoded data with the encoded data to generate aggregated data, wherein the reasoning information is generated by the AI ​​reasoning model based on the aggregated data, and Among them, the multiple AI encoder models are jointly trained with the AI ​​encoder model and the AI ​​reasoning model based on the output of the AI ​​teacher model.

4. The method 700 according to claim 3, wherein: The encoded data is quantized by the AI ​​encoder model before being sent to the first device.

5. The method 700 according to any one of claims 1 to 4, wherein: The second device includes a surveillance camera 811 as the at least one sensor, and The task includes detecting at least one of an object and an event observed by the surveillance camera 811 .

6. The method 700 according to any one of claims 1 to 5, wherein: The second device includes a wearable device 821, and The task includes detecting a health event associated with a user wearing the wearable device 821 .

7. The method 700 according to any one of claims 1 to 6, wherein: The second device comprises an Internet of Things (IoT) device, and The encoded data is received using massive machine type communication (mMTC).

8. The method 700 according to any one of claims 1 to 7, wherein: The AI ​​reasoning model includes a first neural network model, and Among them, the AI ​​teacher model includes at least one of a second neural network model, a support vector machine (SVM) model and an integrated model.

9. An electronic device 1000 for managing sensor data, the electronic device 1000 comprising: At least one memory 1030 storing computer readable instructions; as well as At least one processor 1020, the at least one processor 1020 configured to execute the computer readable instructions to: receiving, at a first device, encoded data from a second device separate from the first device, wherein the encoded data is generated based on sensor data collected by at least one sensor included in the second device using an artificial intelligence (AI) encoder model included in the second device, wherein the first device is the electronic device, providing the encoded data to an AI reasoning model to obtain reasoning information, and performing a task based on the inference information, Wherein, the AI ​​encoder model and the AI ​​reasoning model are jointly trained based on the output of the AI ​​teacher model.

10. The electronic device 1000 according to claim 9, wherein: The at least one processor 1020 is further configured to execute the computer readable instructions to: obtaining, at the first device, a plurality of pieces of encoded data from a plurality of second devices separate from the first device, wherein the plurality of pieces of encoded data are generated using a plurality of AI encoder models included in the plurality of second devices, and combining the plurality of pieces of encoded data with the encoded data to generate aggregated data, wherein the reasoning information is generated by the AI ​​reasoning model based on the aggregated data, and Among them, the multiple AI encoder models are jointly trained with the AI ​​encoder model and the AI ​​reasoning model based on the output of the AI ​​teacher model.

11. The electronic device 1000 according to claim 10, wherein: The encoded data is quantized by the AI ​​encoder model before being sent to the first device.

12. The electronic device 1000 according to any one of claims 9 to 11, wherein: The second device includes a surveillance camera 811 as the at least one sensor, and The task includes detecting at least one of an object and an event observed by the surveillance camera 811 .

13. The electronic device according to any one of claims 9 to 12, wherein: The second device includes a wearable device 821, and The task includes detecting a health event associated with a user wearing the wearable device 821 .

14. The electronic device according to any one of claims 9 to 13, wherein: The second device comprises an Internet of Things (IoT) device, and The encoded data is received using massive machine type communication (mMTC).

15. A computer-readable storage medium storing instructions, which, when executed by at least one processor 1020 of an electronic device 1000 for managing sensor data, causes the at least one processor 1020 to implement the method of any one of claims 1 to 8.

Images