SYSTEMS AND METHODS FOR GENERATING PROCESSABLE DATA FOR MACHINE LEARNING APPLICATIONS - Patent application

Abstract

In one embodiment, the method includes: at a server, generating, for each data type in the one or more data types, an instruction schema from a data schema including one or more data types, the instruction schema including one or more instructions to be applied to the data type; for each device in a plurality of devices communicatively coupled to the server, transmitting the instruction schema from the server to the device, receiving the instruction schema at the device, applying, at the device, each instruction in the instruction schema to locally stored raw user data to generate an embedding of processable data, transmitting the embedding from the device to the server, and receiving the embedding from each device at the server.

Description

[Technical field]

[0001] The present disclosure relates to machine learning, and more particularly, but not by way of limitation, to systems and methods for generating processable data for machine learning applications. [Background technology]

[0002] Traditional training of machine learning algorithms involves copying user data from the device where the data is generated to cloud computers that store and process the data, which not only exposes user data to the risk of being corrupted during transmission or storage, but is difficult and expensive for most companies to implement.

[0003] There have been improvements in techniques that attempt to address user privacy as well as the complexity of such setups, which has hindered progress in the field, slowed enterprise adoption of machine learning, and made experimentation very costly and time-consuming.

[0004] Federated learning solves these challenges by allowing models to train in a distributed fashion, whereby devices that originally generated the data can participate in training a global machine learning model by training locally on the data they themselves generated. While this approach has been found to be effective in scenarios where the data is balanced among the participating devices and each device has a sufficient volume of data to impart meaningful learning to the global model, this approach has been found to be ineffective in imbalanced data situations and in situations where a device may have only one data record. For example, a) a device that contains only a single user profile for the purpose of the global model to classify the owner of that profile as a human or a bot, or b) a device that contains a single sentence for the purpose of identifying whether that sentence is humorous or not. In such cases, it is not possible to train a machine learning model on this data record, since the single data record does not impart variety or meaning to the learning model to be inferred.

[0005] While building on research conducted in the areas of federated machine learning and distributed computing, this disclosure provides a practical solution for achieving the goals of protecting data privacy and significantly reducing the complexity of machine learning systems. Summary of the Invention

[0006] The following presents a simplified summary of the general inventive concepts described herein in order to provide a basic understanding of some aspects of the disclosure. This summary is not an extensive overview of the disclosure. It is not intended to limit key or critical elements of the embodiments of the disclosure, nor to define their scope beyond what is expressly or implicitly described in the following description and claims.

[0007] What is needed is a system and method for generating processable data from distributed raw user data for use in Machine Learning (ML) applications.

[0008] According to one aspect, a computer-implemented method for automatically converting raw user data into processable data for data analysis is presented, the computer-implemented method including: generating, at a server, from a data schema including one or more data types, for each data type in the one or more data types, an instruction schema including one or more instructions to be applied to the data type; for each device in a plurality of devices communicatively coupled to the server, sending the instruction schema from the server to the device; receiving the instruction schema at the device; applying, at the device, each instruction in the instruction schema to the locally stored raw user data to generate an embedding of processable data; sending the embedding from the device to the server; and receiving the embedding from each device at the server.

[0009] In one embodiment, each command includes one or more additional parameters required to apply the command to a data type.

[0010] In one embodiment, applying includes executing an executable function corresponding to the instructions using one or more parameters related to the locally stored raw user data, and appending an output of the executable function to the embedding.

[0011] In one embodiment, the method further comprises, prior to the executing step, identifying, on a memory of the device, an executable function corresponding to the instructions.

[0012] In one embodiment, the instructions include an executable function to be performed.

[0013] In one embodiment, one or more labels are affixed to the implant by the device.

[0014] In one embodiment, at least two of the one or more instructions are chain instructions, each of which is to be applied in a sequence, with the output of a given chain instruction being used as an input for the next chain instruction in the sequence, and the last chain instruction in the sequence generating an embedding.

[0015] In one embodiment, multiple embeddings are generated by the chain instructions, and a final chain instruction is directed to averaging corresponding data types among the multiple embeddings.

[0016] In one embodiment, the method further comprises performing, at the server, a data analysis task on the received embedded processable data.

[0017] In one embodiment, at least some of the instructions are directed to reducing the accuracy of raw user data to make it more difficult to extract private information therefrom.

[0018] In one embodiment, the data analysis task includes a clustering analysis or a similarity test.

[0019] In one embodiment, the data analysis task is a machine learning training task.

[0020] In one embodiment, the machine learning training task uses at least one of supervised learning or unsupervised learning.

[0021] In one embodiment, the training task is executed only each time a specified number of embeddings are received from one or more devices.

[0022] In one embodiment, upon receiving another embedding, the previous training task is resumed.

[0023] According to another aspect, there is provided a system for converting raw user data into processable data for data analysis, the system comprising: a server; a memory for storing a data schema comprising one or more data types; a networking module communicatively coupled to a network; and a processor communicatively coupled to the memory and the networking module and operable to generate, for each data type in the one or more data types, from the data schema, an instruction schema comprising one or more instructions to be applied to the data type; and a plurality of devices, each device comprising a memory, a networking module communicatively coupled to the server via the network, and a processor communicatively coupled to the memory and the networking module, operable to receive the instruction schema from the server via the network, apply each instruction in the instruction schema to raw user data stored in the memory of the device to generate an embedding of processable data, and transmit the embedding via the network to the server, wherein the server is further configured to receive each embedding from the plurality of devices and store it in the memory of the server.

[0024] In one embodiment, each command includes one or more additional parameters required to apply the command to a data type.

[0025] In one embodiment, each of the plurality of devices is configured to apply each instruction by executing an executable function corresponding to the instruction using one or more parameters on the raw user data and adding an output of the executable function to an embedding.

[0026] In one embodiment, the server is further configured to perform machine learning training tasks on the received embedded processable data.

[0027] According to another aspect, a non-transitory computer-readable storage medium is provided that includes instructions that, when processed by a device communicatively coupled to a server via a network, configure the device to perform the steps of receiving from the server via the network an instruction schema including, for each data type in one or more data types of a data schema, one or more instructions to be applied to the data type; applying each instruction in the instruction schema to locally stored raw user data to generate an embedding of processable data; and transmitting the embedding via the network to the server.

[0028] The above and additional aspects and embodiments of the present disclosure will become apparent to those of ordinary skill in the art in light of the detailed description of various embodiments and / or aspects, which are provided with reference to the drawings, a brief description of which is given below.

[0029] To easily identify the description of any particular element or act, the most significant digit or digits in a reference number refer to the figure number in which that element is first introduced. [Brief description of the drawings]

[0030] [Figure 1] 1 is a schematic diagram of a system for generating processable data from distributed raw user data, according to one embodiment. [Diagram 2] FIG. 1 is a schematic diagram illustrating an example of a data schema according to one embodiment. [Diagram 3] FIG. 2 is a schematic diagram illustrating an example of an instruction schema according to one embodiment. [Figure 4] 1 is a process flow diagram illustrating a method for generating processable data from distributed raw user data, according to one embodiment. [Diagram 5] 5 is a schematic diagram illustrating some method steps of the method of FIG. 4 according to one embodiment. [Figure 6] FIG. 2 is a schematic diagram illustrating an example of raw user data according to one embodiment. [Figure 7] 5 is a process flow diagram illustrating an exemplary implementation of method steps of the method of FIG. 4, according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] The present disclosure is directed to systems and methods, according to different embodiments, that provide a mechanism for generating processable data from raw user data locally on distributed networked devices where the raw user data is stored. The processable data has a useful representation of the raw user data that can be easily used by machine learning (ML) algorithms, etc., trained on a remote server. By processing the raw user data locally on each networked device and transmitting the processable data (e.g., data that can be used for further data analysis or ML processes) to the remote server, the storage and processing requirements on the server (i.e., in the cloud) itself are significantly reduced, thereby allowing the server to focus on operating the final step of training the ML algorithm.

[0032] FIG. 1 is a schematic diagram of an exemplary system 100 comprising a server 106 and a plurality of user devices 104 (shown here, for illustrative purposes only, as devices 108a-c) communicatively coupled to one another via a network 110. The user devices 104 may be any type of computing device known in the art, including, but not limited to, a personal computer, a smartphone, a tablet, a smartwatch, and the like. In some embodiments, the server 106 may be a single computer or a virtual server configured to provide software services remotely to the devices 104 "in the cloud." The server 106 and the devices 104 each comprise a memory, a network adapter, and a processor communicatively coupled to the memory and the network adapter. The network 110 may be any type of public or private network, so long as it allows the devices 104 and the server 106 to exchange information. In some embodiments, the devices 104 and the server 106 may communicate over the network 110 using one or more cryptographic processes.

[0033] The server 106 typically includes a data schema 102 stored thereon, which is used to generate an instruction schema 104, as described below. While the data schema 102 typically includes a description of data only, the instruction schema 104 includes instructions in the form of a sequence of operations that can be generated by each of the devices 104 and applied to corresponding raw user data 112 stored on each of the devices 104.

[0034] 2 illustrates an example data schema 102 that includes three fields. The data types or descriptors 202 in the data schema 102, and other elements derived therefrom, are merely examples used to illustrate different embodiments of the present disclosure, and one of ordinary skill in the art will appreciate that any number or type of data may be included in the data schema, without limitation.

[0035] Figure 3 shows an example of the contents of an instruction schema 104 corresponding to the data schema 102 of Figure 2. For example, an instruction 302 generated based on the data type "text data" of the data schema 102 is "Count_digits" which counts the number of digits in a given text and returns the sum.

[0036] 4-6, and according to one exemplary embodiment, a method for generating processable data from raw data stored on a user device, generally referenced by the numeral 400, will now be described. FIG. 5 is a schematic diagram used to illustrate various method steps of FIG. 4. FIG. 5 illustrates an exemplary user device 502 in communication with a server 106. The method 400 starts at step 402 and then proceeds to step 404, where an instruction schema 104 is generated on the server 106 from the data schema 102, as explained above. In step 406, the instruction schema 104 is transmitted from the server 106 to each device connected to the server 106, here the user device 502. In particular, in some embodiments, the server 106 itself transmits the instruction schema 104 to the user device 502, or in other embodiments, the user device 502 may request or retrieve the instruction schema 104 from the server 106. In step 408, each device (here user device 502) applies the instructions in the instruction schema 104 to the raw data 112 stored on each device to generate processable data in the form of embedding 504 from the raw data 112.

[0037] In some embodiments, embedding 504 is an array that is the result of executing all instructions provided in instruction schema 104 against corresponding target elements of instruction schema 104 in raw data 112 stored on user device 502. Thus, in some embodiments, the size of embedding 504 is expected to be the size of the array in instruction schema 104.

[0038] 3, an instruction schema 104 with an array size of 3 results in an embedding 504 of size 3. Thus, in this example, and with reference to the exemplary raw data 112 of FIG. 6, the embedding 504 may look like [4, 22, 0], where the first number in the array refers to the total number of digits in the username (e.g., the username 602 in the exemplary raw data 112 is "Pe1r3s4o4n", which contains four digits), the second number in the embedding refers to the age of the user (derived from the exemplary date of birth "21-10-1999"), and the third number refers to a first category value of a "personal" email address for the email address (e.g., the address "xyz@abc.com").

[0039] In some embodiments, each command sent by the server 106 may include any additional parameters required to allow the command to be fully executed. For example, the command "age" may have a parameter that allows "age" to be calculated in "months" or "years". Thus, an age of 2 years is equal to 24 months, in which case the command schema 104 provides an additional parameter that instructs the user device 502 to calculate the age in months or years.

[0040] In some embodiments, the embedding 504 may be a higher dimensional array based on the complexity of instructions and their outputs, as well as tensors.

[0041] In some embodiments, the instructions in the instruction schema 104 may be concatenated, and the output of one instruction may form the input to the next instruction. In such cases, the output of the last instruction in the chain occurs in the final embedding 504. For example, the instruction "age" may be followed by an instruction to calculate what age group the user belongs to, so that the output of "30" may be "3," referring to the third age group.

[0042] In step 410, the embeddings 504 (from each device) are then sent back to the server 106, which in step 412 uses the received embeddings to train the target machine learning algorithm. The systems and methods described herein may be used with any machine learning model known in the art. Additionally, different machine learning training methods may be used without exception. For example, in some embodiments, the training task may rely on supervised or unsupervised learning methods or models. The method ends in step 414.

[0043] In some embodiments, if a label is needed for training, each device 108 can attach the label to the embedding as the last number in an array.

[0044] In some embodiments, ML training is continuous and may not require waiting for all devices to send their contributions in order to begin training. Training can occur for each batch of new embeddings received (e.g., whenever 500 new embeddings are received, training can begin starting from the last saved training or any checkpoint of the desired model).

[0045] In some embodiments, instructions can be refined over time on the same data set to improve the accuracy of the model and condense the embeddings to only useful information. This can be done by applying feature importance techniques to analyze which instructions were useful for training the model and which were not.

[0046] In some embodiments, a pre-programmed library (Software Development Kit (SDK)) is installed on the device 108 receiving the instruction schema 104. This library can parse the instruction schema 104 and map it to pre-programmed instructions in the SDK. FIG. 7 illustrates an exemplary embodiment of method step 408 of the method 400 described above. In FIG. 7, at step 702, the device receives the instruction schema 104 from the server 106. At step 704, the device parses the instructions in the instruction schema 104 and at step 706, identifies a local function in the SDK that can execute the instructions. At step 708, the local function is applied to the raw data 112 (if any) with the data parameters specified in the instructions, and at step 710, the result is attached to the embedding 504. For example, the username is passed to a digit count function to generate an output of the function that is a number, and this value is attached to the embedding 404.

[0047] In some embodiments, the instructions can be written in any format that can be sent by both the SDK and the server and is a parable. Examples of these formats are XML, JSON, binary, or plain text.

[0048] In some embodiments, the instructions may be sent as instruction definitions that the SDK can convert into functions, as shown in the examples of Figures 4 and 7. However, in some embodiments, the instructions may also be sent as executable code sent over the network 110 that the SDK can execute. In some embodiments, the latter may require validation work to ensure that it is not misused to leak private information about the data.

[0049] In some embodiments, the instructions can be designed to ensure that private information cannot be parsed from the data by reducing the precision of the instructions, for example by using "age group" instead of "age" or by reducing the number of precise characteristics that can identify a user.

[0050] In some embodiments, concatenating instructions allows additional instructions to be applied across embeddings. For example, it is possible to average multiple embeddings generated by a schema on a device, and in order to execute such instructions, the device must store a version of the embedding locally. This case is particularly useful for scenarios where an embedding may represent content with which a user of the device interacts, and thus an embedding is generated for every piece of content with which the user interacts, and thus the instructions may average all embeddings together to generate one embedding that may represent the user's interaction.

[0051] In some embodiments, it may be possible to use the instructions to generate useful labels about the data, such as encoding interactions a user may have with content on a device, to serve as labels for training a system, such as a recommender system.

[0052] In some embodiments, the embedding 504 may be further optimized or refined on the server 106 .

[0053] In some embodiments, the generated embeddings 504 can be used for purposes other than machine learning, such as performing clustering or similarity tests of such embeddings to identify the proximity of some data to other embeddings collected from other devices. An example of this could be calculating the proximity of a user behavior encoded by an embedding to another user behavior encoded using the same instruction schema.

[0054] Although the algorithms described above, including those related to the flow charts above, have been described separately, it is to be understood that any two or more of the algorithms disclosed herein may be combined in any combination. Any of the methods, algorithms, implementations, or procedures described herein may include machine-readable instructions for execution by (a) a processor, (b) a controller, and / or (c) any other suitable processing device. Those skilled in the art will readily appreciate that any algorithm, software, or method disclosed herein may be implemented in software stored on a non-transitory tangible medium, such as, for example, a flash memory, a CD-ROM, a floppy disk, a hard drive, a digital versatile disk (DVD), or other memory device, although the entire algorithm and / or parts thereof may alternatively be executed by a device other than a controller and / or may be implemented in firmware or dedicated hardware in a well-known manner (e.g., it may be implemented by an application specific integrated circuit (ASIC), a programmable logic device (PLD), a field-programmable logic device (FPLD), discrete logic, etc.). Also, some or all of the machine-readable instructions depicted in any flowcharts set forth herein may be implemented manually, as opposed to automatically, by a controller, processor, or similar computing device or machine. Additionally, although particular algorithms have been described with reference to the flowcharts set forth herein, those skilled in the art will readily appreciate that many other ways of implementing the example machine-readable instructions may alternatively be used. For example, the order of execution of the blocks may be changed, and / or some of the described blocks may be changed, eliminated, or combined.

[0055] It should be noted that the algorithms shown and described herein as having various modules performing certain functions interact with each other. It should be understood that these modules are only separated based on their functionality for the purposes of explanation and represent computer hardware and / or executable software code stored in a computer readable medium for execution on suitable computing hardware. The various functions of the different modules and units can be combined or separated as described above as modules in any manner, as hardware and / or as software stored in a non-transitory computer readable medium, and can be used separately or in combination.

Claims

1. 1. A computer-implemented method for automatically converting distributed raw user data into processable data for data analysis, comprising: generating, at a server, from a data schema including one or more data types, for each data type in the one or more data types, an instruction schema including one or more instructions to be applied to the data type; For each device in a plurality of devices communicatively coupled to the server, transmitting the instruction schema from the server to the device; receiving the instruction schema at the device; applying, at said device, each instruction in said instruction schema to locally stored raw user data to generate an embedding of processable data; transmitting the embedding from the device to the server; receiving, at the server, the embedding from each device; 4. A computer-implemented method comprising:

2. 2. The method of claim 1, wherein each of the instructions includes one or more additional parameters required to apply the instruction to the data type.

3. The application of performing an executable function corresponding to said instruction using said one or more parameters on said locally stored raw user data; adding an output of the executable function to the embedding; The method of claim 2 , comprising:

4. Prior to the performing step, identifying the executable function corresponding to the instruction in a memory of the device; The method of claim 3 further comprising:

5. The method of claim 3 , wherein the instructions comprise the executable function to be performed.

6. The method of claim 1 , wherein one or more labels are affixed to the implant by the device.

7. 2. The method of claim 1 , wherein at least two of the one or more instructions are chain instructions, each of the chain instructions being to be applied in a sequence, the output of a given chain instruction being used as an input for a next chain instruction in the sequence, and a last chain instruction in the sequence generating the embedding.

8. 7. The method of claim 6, wherein multiple paddings are generated by said chain instructions, and said final chain instruction is directed to averaging said corresponding data types among said multiple paddings.

9. performing, on the server, data analysis tasks on the received embedded processable data. The method of claim 1 further comprising:

10. 2. The method of claim 1, wherein at least some of the instructions are directed to reducing accuracy of the raw user data to make it more difficult to extract private information therefrom.

11. The method of claim 9 , wherein the data analysis task comprises a clustering analysis or a similarity test.

12. The method of claim 9 , wherein the data analysis task is a machine learning training task.

13. The method of claim 12 , wherein the machine learning training task uses at least one of supervised learning or unsupervised learning.

14. The method of claim 12 , wherein the training task is performed only each time a specified number of embeddings are received from the one or more devices.

15. The method of claim 12 , wherein upon receiving another embedding, a previous training task is resumed.

16. 1. A system for converting raw user data into processable data for data analysis, the system comprising: A server, the server comprising: a memory for storing a data schema including one or more data types; a networking module communicatively coupled to the network; a processor communicatively coupled to the memory and to the networking module, the processor operable to generate from the data schema, for each data type in the one or more data types, an instruction schema including one or more instructions to be applied to the data type; a plurality of devices, each device comprising: a memory; a networking module communicatively coupled to a server over the network; and a processor communicatively coupled to the memory and the networking module; receiving the instruction schema from the server over the network; applying each instruction in the instruction schema to raw user data stored in the memory of the device to generate an embedding of processable data; transmitting said embedding over said network to said server; A plurality of devices operable to Equipped with The system, wherein the server is further configured to receive each embedding from the plurality of devices and store it in the memory of the server.

17. 17. The method of claim 16, wherein each of the instructions includes one or more additional parameters required to apply the instruction to the data type.

18. Each of the plurality of devices is performing an executable function corresponding to said instruction on said raw user data using one or more parameters; adding an output of the executable function to the embedding; 20. The method of claim 17, configured to apply each instruction by:

19. 20. The system of claim 16, wherein the server is further configured to perform machine learning training tasks on the received embedded processable data.

20. When processed by a device communicatively coupled to a server via a network, the device receiving from the server over the network an instruction schema including, for each data type in the one or more data types of the data schema, one or more instructions to be applied to the data type; applying each instruction in the instruction schema to locally stored raw user data to generate an embedding of processable data; transmitting said embedding over said network to said server; 16. A non-transitory computer-readable storage medium comprising instructions configured to execute:

Images