Distributed machine learning model

Abstract

A method comprising, by a first computer equipment: obtaining an input data point comprising a set of values, each value being a value of a different element of an input feature vector; inputting the input data point to a first machine learning model on the first computer equipment to generate at least one associated output label based on the input data point; sending a partial data point to a second computer equipment, the partial data point comprising values of only a portion of the feature vector; and sending the associated label to the second computer equipment in association with the partial data point, thereby causing the second computer equipment to train a second machine learning model on the second computer equipment based on the sent partial and associated label.

Description

Background Technology

[0001] Machine learning refers to the process of training statistical models based on observed data. A well-known form of such machine learning model is the neural network. A neural network consists of multiple nodes interconnected by links (sometimes called edges). Nodes and edges are typically implemented in software. The input edges of one or more nodes collectively form the network's input, while the output edges of one or more other nodes collectively form the network's output, and the output edges of individual nodes within the network form the input edges to other nodes. Each node represents a function of its input edges, which is weighted by its respective weights and output at its output edges. The weights can be progressively adjusted based on a set of empirical data (training data) to tend towards a state where the network will output the desired value for a given input.

[0002] Figure 1A A simplified representation of the example neural network 101 is given by way of illustration. The example neural network includes multiple layers 102 of nodes 104: an input layer 102i, one or more hidden layers 102h, and an output layer 102o (although this hierarchical structure is not required). In practice, there may be many nodes in each layer, but for simplicity, only a few nodes are illustrated. Each node 104 is configured to generate an output by performing a function on the values ​​input to that node. The inputs to one or more nodes together form the input to the neural network, the outputs of some nodes form the inputs to other nodes, and the outputs of one or more nodes together form the output of the network.

[0003] At some or all nodes of the network, the inputs to that node are weighted by their respective weights. Weights can define the connectivity between nodes in a given layer and nodes in the next layer of the neural network. Weights can be in the form of a single scalar value or can be modeled as a probability distribution. When weights are defined by a distribution (as in a Bayesian model), the neural network can be fully probabilistic and capture the concept of uncertainty. The values ​​of the connections between nodes can also be modeled as distributions. This is in... Figure 1B The distribution is illustrated schematically. The distribution can be parameterized using parameters such as mean μ and standard deviation σ or variance σ. 2 It is represented in the form of a sample set or parameter set.

[0004] The network learns by manipulating the data input at the input layer and adjusting the weights imposed by some or all nodes based on that input data. There are different learning methods, but generally, in... Figure 1A The forward propagation, which calculates the overall error from left to right through the network, and in... Figure 1AThe network performs backpropagation of the error from right to left. In the next cycle, each node considers the backpropagation error and produces a revised set of weights. In this way, the network can be trained to perform its desired operation.

[0005] The input to a network is typically a vector, where each element represents a distinct corresponding feature. For example, in image recognition, the elements of this feature vector could represent different pixel values, or in medical applications, different features could represent different symptoms or patient questionnaire responses. The network's output can be a scalar or a vector. The output can be referred to as a label, such as a classification for whether an object (like an elephant) has been identified in an image, or a patient diagnosis in a medical example.

[0006] Figure 1C A simple setup is shown where a neural network is configured to generate labels based on input feature vectors. During the training phase, empirical data comprising a large number of input data points X is provided to the neural network, each data point consisting of a set of example values ​​of a feature vector labeled with its corresponding label Y.

[0007] The label Y can be a single scalar value (e.g., representing elephant or non-elephant) or a vector (e.g., a one-hot vector whose elements represent different possible classification outcomes, such as elephant, hippopotamus, rhinoceros, etc.). Possible label values ​​can be binary or soft values ​​representing percentage probabilities. On many example data points, the learning algorithm adjusts the weights to reduce the overall error between the labeled classification and the classification predicted by the network. Once trained with an appropriate number of data points, unlabeled feature vectors can be fed into the neural network, and the network can predict classification values ​​based on the input feature values ​​and the adjusted weights.

[0008] This type of training is sometimes referred to as a supervised method. Other methods are also possible, such as reinforcement learning, where each data point in the input training data is initially unlabeled. Instead, the learning algorithm starts by guessing the corresponding label for each point and is then told whether it is correct, thus gradually adjusting the weights with each such feedback. Another example is an unsupervised method, where the input data points are not labeled at all, but rather the learning algorithm infers their own structure from the training data. The term "training" used in this article is not necessarily specifically limited to supervised, reinforcement, or unsupervised methods.

[0009] Machine learning models can also be formed from more than one component neural network. An example of this is an autoencoder, such as a variational autoencoder (VAE). In an autoencoder, the encoder network is arranged to process the observed input vector X. oThe input vector is encoded as a latent vector Z, and the decoder network is arranged to decode the latent vector back into the real-world feature space of the input vector. The actual input vector X... o The version of the input vector predicted by the decoder The differences between the input and decoder are used to adjust the weights of the encoder and decoder to minimize a measure of overall difference, for example, based on an evidence lower bound (ELBO) function. The latent vector Z can be considered a compressed form of information in the input feature space. In a variational autoencoder (VAE), each element of the latent vector Z is modeled as a probability or statistical distribution, such as a Gaussian distribution. In this case, for each element of Z, the encoder learns one or more parameters of the distribution, such as the center point and a measure of the spread. For example, the center point could be the mean, and the spread could be the variance or standard deviation. The values ​​of the elements input to the decoder are then randomly sampled from the learned distribution. Once trained, the autoencoder can be used to analyze subsequently observed feature vectors X. o The missing values ​​are estimated in the middle. Alternatively or additionally, a third network can be trained to predict the classification Y from the latent vectors, and then, once trained, it can be used to predict the classification of subsequent unlabeled observations.

[0010] Machine learning models are known to be implemented on server systems (e.g., cloud server systems comprising multiple server units located in different geographical locations). Different client devices of different users can send data captured from their devices to the server system to train server-side models.

[0011] Recently, there has been interest in adopting a more distributed approach, thereby implementing local machine learning models on each user device and a central model on a server system.

[0012] One problem with implementing this model on server systems such as the cloud is the upload bandwidth required to transfer large amounts of data, such as audio and image data, from user devices to the cloud. Another issue is user privacy if data captured from user devices is uploaded directly to the cloud.

[0013] One approach proposed to address these issues is called "joint learning." Joint learning is a form of distributed approach where a neural network model is implemented again on each of multiple client devices and server systems. However, instead of sharing the actual training data among the models, each model on the client device and cloud system is trained based on its own local data. Furthermore, the model parameters (i.e., the weights of the neural network) are shared between the client and cloud sides, and algorithms are applied to interpolate or blend the weights of different component models to share the knowledge accumulated by the different models, without sharing the actual input data. Summary of the Invention

[0014] However, one problem with joint learning is that it requires all models to have the same basic model architecture. Some joint learning algorithms claim to allow heterogeneous models, but because they are based on shared model parameters, the models must still share a basic common architecture to at least some extent. Other truly heterogeneous approaches require sharing the complete training dataset from the client to the server side.

[0015] According to one aspect disclosed herein, a method performed by a first computer device is provided. The method includes: obtaining input data points comprising a set of values, each value being a value of a distinct element of an input feature vector, the elements of the input feature vector comprising a plurality of subsets of elements, each subset comprising one or more of the elements of the feature vector. The method further includes: inputting the input data points to a first machine learning model on the first computer device to generate at least one associated output label based on the input data points. The method further includes: sending partial data points to a second computer device, the partial data points comprising only a portion of the values ​​of the feature vector, the portion comprising one or more of the elements in the subset but excluding one or more other subsets of the elements; and sending the associated label to the second computer device associated with the partial data points, thereby causing the second computer device to train a second machine learning model on the second computer device based on the partial data points and the associated label.

[0016] In this embodiment, the first computer device may be a client device, while the second computer device may be a server system (e.g., a cloud server system comprising multiple server units located in different physical locations).

[0017] Because the client-side and server-side models (or more generally, the first and second models) communicate using partial input data points plus output labels, this advantageously avoids the need to share the complete feature vector from the client, while also enabling a completely model-agnostic approach to distributed machine learning. In other words, because communication between models occurs only with respect to feature data and labels (the model's input and output), not model parameters (the model's internal structure), the disclosed approach can work between any two machine learning models, regardless of their model architecture. For example, these might differ in the number of weights or nodes, the interconnection structure between nodes, or even the type of machine learning model (e.g., one is a VAE, and the other is not; or one is a CNN, and the other is an RNN; or even one includes a neural network while the other is formed by a machine learning model rather than a neural network, etc.). However, if some selected data from the feature vector is sensitive for privacy reasons, or if uploading it would be cumbersome due to bandwidth considerations, or if it cannot be accessed by the server side for some other reason, it can still be omitted from the data sent to the server. For example, in machine learning applications, the user-side model can use both audio and video to add lip reading or facial expressions to inform learning and prediction. However, users can avoid the need to upload videos and instead upload only audio and labels (text in this case) generated by their local model. The server-side model can then learn based on the audio and the provided labels. For example, this could be used to progressively update the training of the server-side model on-site based on additional client-side insights.

[0018] This disclosure is provided to introduce, in a simplified form, a selection of concepts also described in the detailed description. This disclosure is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. The claimed subject matter is also not limited to the implementation of solutions to any or all of the shortcomings indicated herein. Attached Figure Description

[0019] To aid in understanding the embodiments of this disclosure and to illustrate how these embodiments can be implemented, reference is made to the accompanying drawings by way of example only, in which:

[0020] Figure 1A This is a schematic diagram of a neural network.

[0021] Figure 1B This is a schematic diagram of the nodes in a Bayesian neural network.

[0022] Figure 1C This is a schematic diagram of a neural network configured to predict classification based on input feature vectors.

[0023] Figure 2This is a schematic diagram of a computing device used to implement a neural network.

[0024] Figure 3 The illustration illustrates a dataset comprising multiple data points, each data point including one or more values.

[0025] Figure 4 An example of a distributed arrangement of computer equipment connected via a communication network is illustrated schematically, and

[0026] Figure 5 This is a schematic flowchart illustrating a method for operating a distributed machine learning model according to various embodiments disclosed herein. Detailed Implementation

[0027] For example, consider an arrangement where one or more individual client devices (e.g., user devices such as PCs, tablets, and smartphones) connect to a cloud server system. The cloud server system has access to vast amounts of data and also possesses powerful AI processing capabilities. Client devices, such as user devices and edge devices, can use the cloud as a label provider to train local personalized models.

[0028] However, this paper recognizes that client devices may have one or more additional input channels (e.g., cameras) that are inaccessible to the cloud. In such cases, customers can use these channels to assist in validating AI models. By combining the cloud as an oracle with client devices providing additional validation, a hybrid cloud-client machine learning approach will produce superior AI models while enabling a better user experience on client devices using locally customized models.

[0029] In each embodiment, the complete model is not sent to the client device because the client device may not have sufficient computing power to run the cloud-generated model. If the complete model were to be sent, limitations would be placed on the cloud model and its quality (it would need to be small enough to run on client devices such as user devices or edge devices).

[0030] To alleviate the limitations of both parties on quality and computing power, in various embodiments, the client device may have a different model design or architecture compared to the cloud model, wherein the client model is a lighter model trained on local data and labels retrieved by sending local data to a powerful (and very heavyweight) cloud model used to generate labels.

[0031] In this way, local models can be personalized and lightweight (solving personalized problems can be done in lightweight models), while cloud models can be of very high quality without imposing any computational constraints (and can leverage massive amounts of data and advanced non-edge research techniques to tackle larger, more generalized problems). These two approaches differ significantly in terms of computation, requirements, and potential quality (e.g., not limiting the cloud to user device or edge-sized models, but using it as a labeled oracle).

[0032] Furthermore, in the disclosed method, the client can use additional data (quality video, sensitive local data about the user's personal information) that is easily retrieved locally but cannot be sent to the cloud or is insecure to send to the cloud. This data can be used to provide further validation for the server-side model (e.g., the cloud). Additionally, this extra data can be used to personalize the model locally on the client device without imposing model requirements regarding model architecture and model inputs on either side (client and / or cloud).

[0033] This allows edge or user devices to power the cloud with locally generated insights, which are then transferred to the cloud as improved, personalized labels to train a more powerful cloud engine, without imposing restrictions on the input or other aspects of the model design on either side. The cloud can also upgrade its model to a more advanced one without imposing any update requirements or computational limitations on any device, as each device will enjoy improved label retrieval without needing to change its local model architecture accordingly.

[0034] In this embodiment, different client devices (e.g., different user devices or different edge devices) may also have different model designs; whereas in the previous “joint learning” approach to distributed machine learning models, all devices shared the same underlying model design with each other and the cloud, which imposed limitations on model design (even with a powerful cloud, the model could not exceed the minimum device capabilities).

[0035] Figure 2 This is a high-level schematic block diagram of a machine learning (ML) algorithm 206. The machine learning algorithm 206 includes a machine learning model 208 and a learning function 209. The machine learning model 208 includes multiple parameters that can be adjusted based on input data until the model 208 produces a desired or anticipated output (within some uncertainty or error range) for a given input data point. In an embodiment, the machine learning model 208 includes one or more neural networks, in which case the parameters may be referred to as weights (not necessarily weights in the multiplicative sense, although this is certainly a possibility). The model 208 may include, for example,... Figure 1AThe illustration shows a single neural network, or a more complex network structure composed of multiple component neural networks, such as an autoencoder like a variational autoencoder (VAE).

[0036] A machine learning model is a statistical model that is trained on observed training data to produce a desired or anticipated output state for a given input state. Typically, a machine learning model 208 can include a neural network, but this is not the only possibility, and other forms of machine learning models are also known, such as alternative forms of Bayesian probabilistic models, meta-learning models, or clustering algorithms.

[0037] Machine learning algorithm 206 also includes a learning function 209, which is arranged to adjust the parameters of machine learning model 208 based on a learning process based on a set of training data. For example, in a neural network, model 208 includes a network (graph) of nodes, each node being parameterized by one or more weights that are progressively adjusted based on the training data until the output state of the graph best fits the training data to a given set of input states.

[0038] Typically, the machine learning algorithm 206 can be implemented on any form of computing equipment; for example, computing equipment including one or more user terminals, such as desktop computers, laptop computers, tablets, smartphones, wearable smart devices such as smartwatches, or onboard computers in vehicles such as automobiles; and / or server systems including one or more physical server units located in one or more geographic locations (e.g., data centers). Distributed or “cloud” computing technologies are known in the art if desired.

[0039] Machine learning algorithm 206 is implemented in the form of software (code) implemented on computer-readable storage and running on a processing device of a corresponding computer apparatus. The computer apparatus in question may include one or more general-purpose processors, such as a central processing unit (CPU); and / or one or more working accelerator coprocessors, such as a graphics processing unit (GPU), a machine learning accelerator processor, a digital signal processor (DSP), and / or a cryptographic processor. Any such processor may take the form of a Boolean processor based on classical transistors, or another form such as a quantum processor or a biological processor. One or more processors may be implemented on one or more computer terminals or units located at one or more geographical locations. The storage (i.e., memory) on which the code is stored may include one or more memory devices, such as a hard disk drive, a removable storage device, ROM (read-only memory), or RAM (random access memory). The memory may take the form of one or more memory media, such as electronic media such as EEPROM, flash memory, or static or dynamic RAM; or magnetic media such as a disk or magnetic tape; or optical media such as an optical disc or quartz glass. The memory may be implemented on one or more computer terminals or units located at one or more geographical locations.

[0040] The learning function 209 is configured to collect observations (i.e., observed values) of various features in the input feature space. For example, it may be configured to collect input from one or more users via a UI front end, such as a touchscreen, click interface, text interface, voice interface, etc.; or automatically collect data from unmanned devices such as sensor devices. The learning function 209 can then train model 208 based on the collected observations (i.e., data).

[0041] Figure 1A The principle behind a neural network is illustrated. A neural network 101 comprises a graph of interconnected nodes 104 and edges 106 connecting the nodes, all implemented in software. Each node 104 has one or more input edges and one or more output edges, wherein at least some nodes 104 have multiple input edges per node, and at least some nodes 104 have multiple output edges per node. The input edges of one or more nodes 104 form the overall input 108i of the graph (typically an input vector, i.e., multiple input edges exist). The output edges of one or more nodes 104 form the overall output 108o of the graph (which may be an output vector in the case of multiple output edges). Furthermore, the output edges of at least some nodes 104 form the input edges of at least some other nodes 104.

[0042] Each node 104 represents a function of the input values ​​received at its input edges 106i, the output of which is output at the corresponding output edges 106o of the node 104, such that the values ​​output at the output edges 106o of the node 104 depend on the corresponding input values ​​according to the corresponding function. The function of each node 104 is also parameterized by one or more corresponding parameters w, sometimes referred to as weights (not necessarily in the sense of multiplicative weights, although this is certainly a possibility). Therefore, the relationship between the values ​​of the inputs 106i and outputs 106o of each node 104 depends on the corresponding function of the node and its corresponding weights.

[0043] Each weight can simply be a scalar value. Alternatively, as... Figure 1B As shown, at some or all of the nodes 104 in network 101, the corresponding weights can be modeled as probability distributions (such as Gaussian distributions). In this case, neural network 101 is sometimes referred to as a Bayesian neural network. Optionally, the input / output values ​​at each edge of some or all of the edges 106 can also be modeled as respective probability distributions. For any given weight or edge, the distribution can be modeled based on a sample set of the distribution or a set of parameters that parameterize the corresponding distribution, for example, specifying a pair of parameters for its center point and width (e.g., based on its mean μ and standard deviation σ or variance σ²). The values ​​of the edges or weights can be random samples from the distribution. Learning the weights can include adjusting one or more parameters for each distribution.

[0044] like Figure 1A As shown, the nodes 104 of the neural network 101 can be arranged into multiple layers, each layer including one or more nodes 104. In a so-called "deep" neural network, the neural network 101 includes an input layer 102i, one or more hidden layers 102h (also called inner layers), and an output layer 102o. The input layer 102i includes one or more input nodes 104i, each hidden layer 102h includes one or more hidden nodes 104h (or inner nodes), and the output layer 102o includes one or more output nodes 104o. For simplicity, Figure 1A Only two hidden layers (102h) are shown, but there may be more hidden layers.

[0045] The different weights of the individual nodes 104 in the neural network 101 can be gradually adjusted based on a set of empirical data (training data) to tend towards a state where the network output 108o will produce the expected value of a given input 108i. For example, the neural network 101 can be trained for an application before being used in a practical application. Training involves inputting empirical data in the form of training data into the input 108i of the graph, and then adjusting the weights w of the nodes 104 based on feedback from the outputs 108o of the graph. The training data includes multiple different input data points, each including a value or value vector corresponding to one or more input edges 108i of the graph 101.

[0046] For example, consider as Figure 1C The simplified example shown illustrates a machine learning model comprising a single neural network 101, arranged to take a feature vector X as its input 108i and output a label Y as its output 108o. The input feature vector X comprises multiple elements x. d Each element represents a different feature d = 0, 1, 2, ..., etc. For example, in an image recognition example, each element of the feature vector X can represent its respective pixel value. For example, one element represents the red channel of pixel (0,0); another element represents the green channel of pixel (0,0); another element represents the blue channel of pixel (0,0); another element represents the red channel of pixel (0,1); and so on. As another example, in the case of using neural networks for medical diagnosis, each element of the feature vector can represent the value of different symptoms of the object, the physical characteristics of the object, or other facts about the object (such as body temperature, blood pressure, etc.).

[0047] Figure 3 This shows an example dataset containing multiple data points i = 0, 1, 2, ..., etc. Each data point i includes the corresponding set of values ​​for a feature vector (where x... id It is the value of the d-th feature in the i-th data point. Input feature vector X i Represents an input observation for a given data point, where any given observation i may or may not include the complete set of values ​​for all elements of the feature vector X. Label Y i This represents the corresponding classification of observation i generated by the network based on the input. In the training data, the label Y... i The observed values ​​can be specified using the observed values ​​of each data point along with the elements of the feature vector. In the subsequent prediction phase, the value of the label Y is predicted by the neural network 101 for use as further input observation X.

[0048] The label Y can be a scalar or a vector. For example, in a simple example of an elephant recognizer, Y could be a single binary value representing whether an image is an elephant or not, or a soft value representing the probability or confidence that an image includes an elephant image. Or similarly, if the neural network 101 is used to test a specific medical condition, Y could be a single binary value representing whether an object has that condition, or a soft value representing the probability or confidence that an object has the condition being tested. As another example, Y could include a “1-hot” vector, where each element represents, for example, a different animal or condition. For example, Y = [1,0,0,…] represents an elephant, Y = [0,1,0,…] represents a hippopotamus, Y = [0,0,1,…] represents a rhinoceros, etc. Or, if soft values ​​are used, Y = [0.81,0.12,0.05,…] means that the confidence that an image includes an elephant image is 81%, the confidence that it includes a hippopotamus image is 12%, the confidence that it includes a rhinoceros image is 5%, and so on.

[0049] During the training phase, the Y-axis of each data point i... i The true value is known. For each training data point i, the ML algorithm 206 measures the resulting output value of the output edge or edge 108o of the graph and uses this feedback to progressively adjust the different weights w of each node 104 so that, over many observed data points, the weights tend to make the output(s) 108i(Y) of graph 101 as close as possible to the actual observed values ​​(for a given overall error metric) in the empirical data across the training inputs. That is, for each input training data, a predetermined training output is compared with the actual observed output of graph 108o. This comparison provides feedback over multiple training data points to progressively adjust the weights w of each node 104 in the graph toward a state where the actual output 108o(Y) of the graph will closely match the expected or anticipated output of a given input 108i. Examples of such feedback techniques include, for example, random backpropagation.

[0050] Once trained, the neural network 101 can then be used to infer the value of the output label 108o(Y) for a given value of the input vector 108i(X), and vice versa.

[0051] Explicit training based on labeled training data is sometimes called a supervised method. Other machine learning methods are also possible. For example, another example is the reinforcement method. In this case, neural network 101 begins by labeling each data point i with a label Y. iThe initial predictions are often inaccurate or nonexistent. After making a prediction for each data point i (or at least some of them), the ML algorithm 206 (e.g., from a human) receives feedback on whether the prediction was correct and uses this feedback to adjust the weights so that it performs better next time. Another example is known as the unsupervised approach. In this case, the ML algorithm receives no labels or feedback and instead infers its own structure from experienced input data.

[0052] Figure 1C A simple example using neural network 101 is shown, where a single neural network is used to predict label Y from input vector X. In some more complex cases, machine learning model 208 may include two or more structures that make up neural network 101, such as VAEs.

[0053] Figure 4 Example arrangements according to various embodiments disclosed herein are illustrated. The arrangement includes multiple computer systems 402. These include at least a server system 402B and one or more client devices 402A. The server system 402B includes one or more server units 403 located at one or more geographical locations. In embodiments, the server system 402 may take the form of a cloud server system comprising multiple server units 403 located at different geographical locations (e.g., in different data centers). Each client device 402A includes one or more units of a client computer. The server units 403 and client devices 402A are interconnected via a communication network 401, which may include one or more constituent networks; for example, a wide area network such as the Internet, a mobile cellular network such as a 3GPP network, a wired local area network (LAN) such as Ethernet, and / or a wireless LAN such as Wi-Fi, Thread, ZigBee, or 6LoWPAN networks.

[0054] In this embodiment, some or all of the client devices in the client devices each include a user device. A given user device may include one or more units of user equipment (i.e., one or more user terminals). Alternatively or additionally, one, some, or all of the client devices 402A may each include an edge device. An edge device is a device that includes one or more edge devices (e.g., access points and / or routers) that provides an entry point to the communication network 401 for one or more user devices. Figure 4 The document shows three user devices 402Ai, 402Aii, and 402Aiii as examples of client devices 402A, but it should be understood that this is merely an example and that other numbers of the same or different types of client devices 402A may be involved in this arrangement.

[0055] Figure 4 The arrangement can be used to implement a distributed machine learning model. The distributed model comprises individual component machine learning models 208B, 208Ai, 208Aii, 208Aii, etc., respectively, on each of the server system 402B and client devices 402Ai, 402Aii, 402Aiii. Each forms part of a corresponding machine learning algorithm 206B, 206Ai, 206Aii, 206Aiii implemented on the corresponding computer equipment. Each algorithm 206 is implemented on the memory of its respective computer equipment and is arranged to run on the corresponding processing device of that computer equipment, as previously referenced. Figure 2 The discussion. Each algorithm 206 can be referenced. Figure 2 The principles described are used for configuration. However, the models on server system 402B and (various) client devices 402A (referred to herein as client-side model 208B and server-side model 208A, respectively) may have different designs from each other. For example, server-side model 208B may be more complex (e.g., more weights, nodes, and / or input edges, and / or a more complex model structure). This will require more memory and / or processing resources from server system 402B to implement. Client-side model 208A may be less complex, but more personalized for the specific application or environment of the corresponding client device 402A. In some cases, the different models 208B and 208A may even be different types, such as one being a convolutional neural network (CNN) and the other a recurrent neural network (RNN), or one including a VAE and the other not including a VAE. Additionally, or alternatively, models 208Ai and 208Aii on two or more different client devices 402Ai and 402Aii may have different model designs, for example, to be customized for two different applications or situations on those different client devices.

[0056] Figure 5 The general method that can be performed according to the various embodiments disclosed herein is illustrated. Steps S10, S40, S50, S60, and S90 form a compositional method that can be performed by client device 402A. By way of description, the method will be described from the perspective of a specific client device 402Ai, but it will be understood that the same method can be implemented on any client device 402A. Steps S20, S30, S70, and S80 form a compositional method that can be performed by server system 402B. Steps S10 to S50 and S80 to S90 are optional.

[0057] The client device 402Ai has specific feature vectors available for collecting data thereon. For example, consider speech-to-text conversion. In this case, the feature vectors might include a first subset of audio samples representing a person's speech and a second subset of video samples, which could include video of a person's lips or face while speaking. Alternatively, or in addition to video, the available feature vectors might also include other elements, such as one or more metadata fields for including proximity information indicating the proximity of the person to the microphone while speaking, or light level information giving information about the light level of the captured video, and so on.

[0058] The data values ​​of the feature vector can be sampled locally on the client device 402Ai, or collected from one or more external sources, or a combination thereof. For example, in the case of speech-to-text conversion, onboard microphones and cameras can be used to capture audio and video, and any proximity or light level information (if used) can be sampled from one or more onboard sensors. As another example, if the feature vector includes medical information, this information can be sampled from one or more internal or external sensors of the client device 402Ai, and / or as survey information input via the user interface of the client device 402Ai. For example, human insight tools (such as allowing users to enter correct / incorrect labels as additional verification input) can be provided. Not only technical sensors can add local insights. Another example of local input could be repeating the same action (e.g., speaking something to voice control) until the correct output is obtained, which can indicate that all previously inserted ambiguous inputs were actually labeled by the same output for training purposes.

[0059] In step S10, the client device 402Ai collects one or more initial data points, each initial data point comprising a set of values ​​for some or all elements of the input feature vector. Each initial data point may be a complete data point including the values ​​of all elements of the feature vector, or it may be a partial data point including the values ​​of only some elements of the feature vector. In various embodiments, each of these initial data points includes only a portion of the feature vector (e.g., including only audio and excluding video or metadata). The client device 402Ai sends the initial data points to the server system 402B. At this stage, the client-side model 208Ai may be untrained or undertrained, and therefore may not be able to generate good labels from the data points themselves.

[0060] In step S20, server system 402B receives initial data points and inputs these data points into server-side model 208B to generate a corresponding label from each data point (e.g., text from a speech sample in the speech-to-text example). Labels can include absolute values ​​or probabilities (soft values), or values ​​and associated probabilities. Server-side model 208B has been pre-trained, for example, based on data from multiple other client devices 402A, and is therefore capable of generating good labels. In step S30, server system 402B sends the generated labels back to client device 402Ai.

[0061] In step S40, labels generated on the server side (e.g., the cloud) are used as a reference to train the local model 208A on the client device 402Ai. Data points plus labels provide labeled training data from which the client-side model 208Ai can be trained. For example, the cloud speech model 208B converts the input audio into text labels, which are then used as training data to train the client-side speech model 208Ai.

[0062] Notice, Figure 5 To illustrate, in reality, server system 402B can send data points, pass data points through server-side model 208B, and return them to client 402Ai to perform training in a continuous or pipelined manner, instead of having to queue up multiple data points before inputting them all through model 208B and then returning the results collectively.

[0063] In step S50, at some point, client device 402Ai detects that its local client-side model 208Ai has reached a certain threshold quality (e.g., cloud quality). Quality can be measured, for example, based on the uncertainty, error, or confidence value output by client-side model 208A. The ability to generate such a measurement of model quality itself is known in the art. Regardless of the measurement method, once client-side model 208A has reached the threshold quality, client device 402Ai can begin locally using its own client-side model 208Ai to generate labels for additional data points, instead of continuing to return to server 402B to obtain labels. Alternatively, client device 402Ai can begin using its own model and continue sending additional data points to server system 402B in parallel to continue training according to steps S10-S40, while making predictions using its own local model 208Ai. Or, in some cases, client device 402Ai may only return to server system 402B to obtain labels under specific circumstances (e.g., if it encounters a word it cannot recognize or cannot recognize with high confidence).

[0064] Note that steps S10-S50 are not required. In other implementations, the client-side model 208Ai can be trained in some other way, for example, it can be a pre-trained model trained offline before being deployed to the client device 402Ai.

[0065] Regardless of how the client-side model 208Ai is initially trained, it can be used to provide validation to aid in training the server-side model 208B, according to the teachings disclosed herein. The client device 402Ai has additional input channels available that are not necessarily provided to the server system 402B but can provide strong quality validation. For example, such data could include camera data, proximity data, distance data, and / or data about the light in the room. In other words, the input feature vector available to model 208Ai on the client device 402Ai includes one or more additional fields (one or more additional subsets of elements) than the vectors available from the client device 402Ai to model 208B on the server system 402B. For example, in the speech-to-text conversion example, a user of the client device 402Ai can capture both the speaker's audio and video. However, for privacy or bandwidth reasons, or both, the user may not want to, or even be allowed to, upload video to the server system 402B and may only upload audio. And / or, metadata or manual data, such as sensor data, may not be accessible to the server system 402B, possibly because it is not configured to access or be able to access such data (e.g., the data is not provided in a standardized form, preventing the server system from collecting metadata from various client devices 402Ai, 402Aii, 402Aiii). Nevertheless, video, sensor data, and / or manual input data may contain strong cues that aid in speech-to-text conversion; for example, lip or facial expressions in video or proximity data from sensors.

[0066] To utilize this additional information, and in accordance with the techniques disclosed herein, the client device 402Ai and server system 402B can be configured to perform the following method steps. These steps occur at least after the initial training phase of the client-side model 208Ai (regardless of how it was initially trained). They can also be performed after the initial training phase of the server-side model 208B and used to improve the training of the server-side model 208. For example, validation can be provided continuously to adapt the training of the server-side model 208B "in-situ," i.e., after the deployment of the real-time use phase of model 208B, in which model 208B is used to provide actual services to users (e.g., via the Internet). However, it is not excluded that the following steps can be used alternatively or additionally as part of the initial training phase of the server-side model 208B.

[0067] In step S60, the client device 402Ai acquires the values ​​of the complete feature vector (i.e., data points with values ​​of all elements of all available fields of the feature vector) and inputs these values ​​through its own local model 208Ai to generate output labels. Labels may include absolute values ​​or probabilities (soft values), or values ​​and associated probabilities. For example, in the speech-to-text example, step S60 may include inputting audio and video data into the client-side model 208A, and possibly any available sensor data or other such metadata, to generate text (output labels) based on all this available information. The output may also include probabilities associated with the predicted text. Additional data on top of the audio provides additional relevant information that can potentially inform text prediction. For example, video may contain motion images of the speaker's lips or face, which will add additional information to the mix to inform text generation.

[0068] Then, the client device 402Ai sends only a partial version of the data points to the server system 402B, which includes the values ​​of only one or some fields of the feature vector, excluding the values ​​of other fields (i.e., only the values ​​of a subset or subset of the elements of the feature vector). For example, it may include audio samples but not video, and / or exclude any sensor data or other such metadata (e.g., manually entered data). However, the client device 402Ai does send the corresponding labels generated by the client device 402A based on the complete feature vector and its own local model 208Ai to the server system 402B.

[0069] In step S70, the server system 402B trains the server-side model 208B, for example, by updating the weights of the neural network based on partial data points and labels received from the client device 402Ai. The partial data points, along with labels, provide labeled training data that can improve the training of the server-side model 208B. Errors and good validations from the client-side model 208Ai can be sent to the cloud as additional learning cycles, which in turn leads to a better cloud model based on client insights into the desired output.

[0070] At a later time, in step S80, the updated server-side model 208B can be used to generate one or more additional labels corresponding to one or more new additional data points (which may also include only one or some fields of the feature vector, similar to the data points sent to the server in step S60). For example, in some cases, in step S90, the method can return to step S10 and repeat steps S10-S40 using the updated version of the server-side model 208B and its output labels as an improved reference. Alternatively or additionally, additional data points can be received as queries from one or more other client devices 402Aii, 402Aiii, etc., so that the server system can generate labels to feed back to those other client devices. For example, this can be used to provide label services to clients 402Aii, 402Aiii that do not have values ​​of the complete feature vector available and / or do not have their own good local models, or to help train their local models (i.e., to perform steps S10-S50 for another client device 402Aii or 402Aiii, etc.).

[0071] While the examples above have already described speech-to-text conversion, it should be understood that this is not limiting. Another example, for instance, is image recognition, such as for autonomous driving, human recognition, or any other application. In this case, some of the data sent to the server system 402B may include image data, while additional data available only to the client device 402Ai might include inertial sensor data from one or more sensors that measure the motion of one or more cameras capturing the image data. This additional data can help, for example, increase insight into when an image is more likely to be blurry.

[0072] Another example is in the medical field, where the output label includes a prediction of the object's condition (e.g., disease). The input feature vector could include medical image data; other measurements obtained from the object's body (e.g., body temperature, blood pressure, etc.); and / or survey data from the object's responses or about the object. It might be desirable to make only some of this information available to the cloud.

[0073] Regardless of the application, note that client device 402A and server system 402B only need to communicate with each other regarding partial input data points and output labels, rather than communicating with each other regarding the parameters of models 208B and 208A as in the case of joint learning. This allows different models to communicate with each other, regardless of their model architecture or design. At the same time, client 402A does not need to share complete feature vectors with server systems (e.g., the cloud). This can be advantageous, for example, for privacy or bandwidth reasons.

[0074] It will be understood that the above embodiments are described by way of example only.

[0075] For example, while the above embodiments have been described based on server-side models with higher complexity than client-side models, this is not necessarily the case for all possible applications. For instance, quantum computers may have models with higher complexity than the cloud. Or in some cases, one model may be more complex in some aspects, such as having only voice input but for a larger network, while another model may be more complex in other aspects, such as having a smaller network but with added video input.

[0076] Furthermore, while the above description has been based on client devices and server systems, the same principles can be extended to any first and second computer devices that implement the first and second machine learning models, respectively. Generally, any two models can collaborate to achieve a higher quality than either model alone, such as two peers, like devices in a smart home. For example, the first device could be a PC, and the second device could be a mobile device such as a smartphone, where the models on the PC and mobile device can have different architectures and / or inputs and can support each other. Other examples could include PCs, televisions, telephones, etc. The cloud is not mandatory, nor is the server-client relationship, as long as the models communicate in terms of feature data and labels to collaborate.

[0077] Furthermore, the disclosed techniques are not limited to use on only one cloud model, but can be used on two or more cloud models; for example, one model for the technical field and another for the general field. Both models may reside in the cloud and may have different inputs and architectures. An example is a voice for general medical dialogue versus a model specifically for medical COVID-19 dialogue. A specific COVID-19 model may include different inputs, such as known symptom fields not present in the general medical model. These two models may differ, but both can reside in the cloud and provide support to and receive validation from clients, as described herein. Additionally, more cloud models may include general (non-medical) dialogue models and / or models for other diseases, such as influenza or cancer models (etc.). All of these may be able to deliver common inputs and / or labels collaboratively (not just at the server / cloud edge).

[0078] In further variations, machine learning models are not limited to neural networks. Other types of machine learning models are also known in the art and can be used for one or both of these models. These other forms of machine learning models include, for example, Markov models, random forests, collaborative filtering, KNN, and other unsupervised clustering methods. If a machine learning model removes architectural constraints and shares training insights among itself, then all machine learning models can share distinct inputs and outputs.

[0079] More generally, according to one aspect disclosed herein, a method is provided performed by a first computer device: obtaining input data points comprising a set of values, each value being the value of a distinct element of an input feature vector, the elements of the input feature vector comprising a plurality of subsets of elements, each subset comprising one or more of the elements of the feature vector; inputting the input data points into a first machine learning model on the first computer device to generate at least one associated output label based on the input data points; sending partial data points to a second computer device, the partial data points comprising only a portion of the values ​​of the feature vector, the portion comprising one or more of the subsets of elements but excluding one or more other subsets of the subsets of elements; and sending the associated label to the second computer device associated with the partial data points, thereby causing the second computer device to train a second machine learning model on the second computer device based on the partial data points and the associated label.

[0080] For example, each machine learning model can include a neural network.

[0081] The acquisition of data points may include capturing values ​​from one or more internal sources of the first computer equipment (e.g., sensors or user interfaces), or receiving values ​​from an external source (e.g., another client device), or a combination thereof.

[0082] In various embodiments, the second model may have higher complexity than the first model. A more complex model will consume more memory and / or processing resources to implement. For example, higher complexity may include more model parameters (e.g., weights in the case of a neural network) and / or more nodes (in the case of a neural network). As another example, the second model may include a more complex model structure than the first model (e.g., a VAE as opposed to a single neural network).

[0083] In various embodiments, the second computer equipment may include a server system, while the first computer equipment may include client devices. The server system may include one or more server units located at one or more geographical locations. In some embodiments, the server system may include a cloud server system comprising multiple server units located in different locations.

[0084] In various embodiments, the client device may include a user device. Alternatively, the client device may be an edge device that provides an entry point between one or more user devices and a communication network that connects the user equipment to the server system.

[0085] In various embodiments, one subset of the plurality of subsets may include an audio field whose values ​​include audio data of a person's speech, and another subset of the plurality of subsets may include a video field whose values ​​include video data of a person's lips or face while the speech is being spoken. In some such embodiments, a first model may be arranged to perform speech-to-text conversion based on an input feature vector, while a second model may be arranged to perform speech-to-text conversion based on that portion of the feature vector, the output label including text. As an example, that portion of the feature vector may include an audio field but not a video field.

[0086] In an embodiment, one subset of the plurality of subsets may include an image field, the value of which includes image data, and another subset of the plurality of subsets includes an inertial sensor data field, the value of which includes inertial sensor data from one or more sensors that measure camera motion while capturing image data. In some such embodiments, a first model may be arranged to perform image recognition to detect objects based on an input feature vector, while a second model may be arranged to perform image recognition based on a portion of the input feature vector, the output label including an indication of the object. As an example, said portion of the feature vector includes an image data field but does not include an inertial sensor data field.

[0087] In various embodiments, the method may further include: during an initial training phase prior to obtaining the input data points, the first computer equipment: obtaining a plurality of initial data points, each initial data point comprising some or all of a corresponding set of values ​​from a subset of elements of the feature vector; sending each of the plurality of initial data points to the second computer equipment, and in response, receiving back associated labels generated by the second model based on the initial data points; and training the first model based on the initial data points and the associated labels received from the second computer equipment. Some or all of the subset may be the portion of the feature vector.

[0088] In a further embodiment, the method may further include, in a subsequent training phase following the sending of the label to the second computer equipment, the first computer equipment sending another data point to the second computer equipment, the other data point comprising some or all of the values ​​in a subset of the elements of the feature vector, and in response, receiving another label generated by the second model based on the other data point; and updating the training of the first model based on the other label and the other data point. Again, some or all of the subset may be the portion of the feature vector.

[0089] According to another statement of the teachings disclosed herein, a method is provided performed by a first computer device: obtaining input data points comprising a set of values ​​of all elements of an input feature vector, each value being the value of a distinct element of the input feature vector, the input feature vector comprising a plurality of fields, each field comprising one or more of the elements of the feature vector; inputting the input data points into a first machine learning model on the first computer device to generate at least one associated output label based on the input data points; sending partial data points to a second computer device, the partial data points comprising only a portion of the values ​​of the feature vector, the portion comprising one or more of the fields of the feature vector but excluding one or more other fields of the fields; and sending the associated label to the second computer device associated with the partial data points, thereby causing the second computer device to train a second machine learning model on the second computer device based on the partial data points and the associated label.

[0090] In various embodiments, one of the plurality of fields may include an audio field, the value of which includes audio data of a person's speech, and another of the plurality of fields may include a video field, the value of which includes video data of the person's lips or face while speaking the speech; wherein the first model may be configured to perform speech-to-text conversion based on the input feature vector, and the second model may be configured to perform speech-to-text conversion based on the portion of the feature vector, the output label including the text; and wherein the portion of the feature vector may include the audio field but not the video field.

[0091] In other embodiments, one of the plurality of fields may include an image field, the value of which includes image data, and another of the plurality of fields may include an inertial sensor data field, the value of which includes inertial sensor data from one or more sensors that measure the motion of the camera while capturing the image data; wherein the first model may be arranged to perform image recognition to detect an object based on the input feature vector, and the second model may be arranged to perform image recognition based on the portion of the input feature vector, the output label including an indication of the object; and wherein the portion of the feature vector may include the image data field but not the inertial sensor data field.

[0092] In various embodiments, the method may include, during an initial training phase prior to obtaining the input data points, the first computer equipment: obtaining a plurality of initial data points, each initial data point comprising a set of corresponding values ​​for some or all of the fields of the feature vector; sending each of the plurality of initial data points to the second computer equipment, and in response, receiving back associated labels generated by the second model based on the initial data points; and training the first model based on the initial data points and the associated labels received from the second computer equipment.

[0093] In various embodiments, the method may include, in a subsequent training phase following the sending of the label to the second computer equipment, the first computer equipment sending another data point to the second computer equipment, the other data point including some or all of the values ​​in the fields of the feature vector, and in response, receiving another label generated by the second model based on the other data point; and updating the training of the first model based on the other label and the other data point.

[0094] In various embodiments, the second model may have higher complexity than the first model, and the more complex model requires more memory and / or processing resources to implement.

[0095] In various embodiments, the second computer equipment may include a server system, while the first computer equipment may include a client device, which may include a user device or edge device that provides an entry point between one or more user devices and a communication network that connects the user devices to the server system.

[0096] According to another aspect disclosed herein, a computer program is provided that implements a method of a client device or first computer apparatus according to any embodiment disclosed herein, which is stored on a computer-readable storage and configured to perform such a method when run on a client device or other such first computer apparatus.

[0097] According to another aspect, a first computer apparatus is provided, comprising: a processing device including one or more processing units; and a memory including one or more memory units, the memory storing a program arranged to run on the processing device; wherein the program is configured to perform a method of a client device or the first computer apparatus according to any embodiment disclosed herein when run on the processing device.

[0098] According to another aspect disclosed herein, a method performed by a second computer device is provided, comprising: receiving partial data points from a first computer device, the partial data points being a partial version of complete data points, the complete data points comprising a set of values, each value being a value of a distinct element of a feature vector, the elements of the feature vector comprising a plurality of subsets of elements, each subset comprising one or more of the elements of the feature vector, wherein the partial data points comprise only a portion of the values ​​of the feature vector, the portion comprising one or more of the subsets of elements but excluding one or more other subsets of the subsets of elements; receiving, in association with the received partial data points, output labels generated by a first machine learning model on the first computer device based on the complete data points; and training a second machine learning model on the second computer device based on the partial data points and the associated labels received from the first computer device.

[0099] In various embodiments, the method may further include, in a subsequent stage following training of the second model by the second computer equipment based on the partial data points and associated labels: generating new labels using the second model for new data points including some or all of the values ​​in the subset of elements of the feature vector, and sending the new labels to the first computer equipment or another computer equipment to train the first model on the first computer equipment or another machine learning model trained on the other computer equipment. Some or all of the subset may be the portion of the feature vector.

[0100] According to another statement of the teachings disclosed herein, a method is provided performed by a second computer device: receiving partial data points from a first computer device, said partial data points being a partial version of complete data points, said complete data points comprising a set of values ​​of all elements of a feature vector, each value being the value of a distinct element of said feature vector, said elements of said feature vector comprising multiple fields, each field comprising one or more of said elements of said feature vector, wherein said partial data points comprise only a portion of the values ​​of said feature vector, said partial data points comprising one or more of said fields of said feature vector but excluding one or more other fields of said fields; receiving, in association with the received partial data points, output labels generated by a first machine learning model on the first computer device based on said complete data points; and training a second machine learning model on the second computer device based on said partial data points and said associated labels received from the first computer device.

[0101] In various embodiments, the method may include, in an initial training phase prior to receiving the partial data points, the second computer equipment: receiving a plurality of initial data points from the first computer equipment, each initial data point comprising a set of some or all corresponding values ​​in the fields of the feature vector; and, in response, generating associated labels by inputting the initial data points into the second model and sending the associated labels back to the first computer equipment, thereby enabling the first computer equipment to train the first model based on the initial data points and the associated labels sent from the second computer equipment.

[0102] In various embodiments, the method may include, in a subsequent stage following training of the second model by the second computer equipment based on the partial data points and associated labels: using the second model to generate new labels for new data points including some or all of the values ​​in the field of the feature vector, and sending the new labels to the first computer equipment or another computer equipment to train the first model on the first computer equipment or another machine learning model trained on the other computer equipment.

[0103] In various embodiments, the method performed by the second computer device may include equivalent steps that supplement the client-side method according to any embodiment disclosed herein.

[0104] According to another aspect disclosed herein, software is provided that implements a method of a server system or second computer apparatus according to any embodiment disclosed herein, which is stored on a computer-readable storage and configured to perform when run on a server system or other such second computer apparatus.

[0105] According to another aspect, a server system or other second computer equipment is provided, which is configured to perform the method of the server or second computer equipment according to any embodiment disclosed herein.

[0106] According to another aspect, a distributed arrangement of computer equipment is provided, the arrangement including a first computer equipment (e.g., a client device) and a second computer equipment (e.g., a server system) in any of the embodiments disclosed herein.

[0107] Once this disclosure is given, other variations or use cases of the disclosed technology may become apparent to those skilled in the art. The scope of this disclosure is not limited to the described embodiments, but is defined only by the appended claims.

Claims

1. A method comprising: equipped with a first computer Obtain input data points, which include a set of values ​​of all elements of an input feature vector, each value being the value of a different element of the input feature vector, and the input feature vector including multiple fields, each field including one or more of the elements of the feature vector; The input data points are fed into a first machine learning model on the first computer equipment to generate at least one associated output label based on the input data points; Send a subset of data points to a second computer device, the subset of data points comprising only a portion of the values ​​of the feature vector, the subset comprising one or more of the fields of the feature vector but excluding one or more other fields of the fields; as well as The associated tag is sent to the second computer equipment associated with the partial data points, thereby enabling the second computer equipment to train a second machine learning model on the second computer equipment based on the partial data points and the associated tag.

2. The method as described in claim 1, characterized in that, One of the plurality of fields includes an audio field, the value of which includes audio data of a person’s speech, and another of the plurality of fields includes a video field, the value of which includes video data of the person’s lips or face while speaking the speech; The first machine learning model is configured to perform speech-to-text conversion based on the input data points, and the second machine learning model is configured to perform speech-to-text conversion based on the partial data points, wherein the output label includes the text; and wherein the partial feature vector includes the audio field but does not include the video field.

3. The method as described in claim 1, characterized in that, One of the plurality of fields includes an image field, the value of which includes image data, and another of the plurality of fields includes an inertial sensor data field, the value of which includes inertial sensor data from one or more sensors that measure the motion of the camera when the image data is captured; The first machine learning model is configured to perform image recognition to detect objects based on the input data points, and the second machine learning model is configured to perform image recognition based on the partial data points, wherein the output label includes an indication of the object; and wherein the partial feature vector includes the image data field but does not include the inertial sensor data field.

4. The method as described in any of the preceding claims, characterized in that, This includes the initial training phase performed by the first computer equipment before obtaining the input data points: Obtain multiple initial data points, each initial data point including a set of some or all of the corresponding values ​​in the fields of the feature vector; Send each of the plurality of initial data points to the second computer device, and in response, receive back the associated label generated by the second machine learning model based on the initial data points; as well as The first machine learning model is trained based on the initial data points and the associated labels received from the second computer equipment.

5. The method as described in claim 4, characterized in that, This includes the subsequent training phase after the first computer equipment sends the tag to the second computer equipment: Send another data point to the second computer device, the other data point including some or all of the values ​​in the fields of the feature vector, and in response, receive back another label generated by the second machine learning model based on the other data point; and The training of the first machine learning model is updated based on the other label and the other data point.

6. A first computer device, comprising: A processing apparatus that includes one or more processing units; as well as A memory comprising one or more memory units, the memory storing a program arranged to run on the processing device; The program is configured to perform the method of any one of claims 1 to 5 when it is run on the processing device.

7. A method comprising being equipped with a second computer: Receiving partial data points from a first computer device, said partial data points being partial versions of complete data points, said complete data points comprising the set of values ​​of all elements of a feature vector, each value being the value of a different element of the feature vector, said elements of the feature vector comprising multiple fields, each field comprising one or more of said elements of the feature vector, wherein said partial data points comprise only a portion of the values ​​of the feature vector, said partial data points comprising one or more of said fields of the feature vector but excluding one or more other fields of said fields; In association with the received partial data points, an output label generated by a first machine learning model on the first computer equipment based on the complete data points is received; as well as A second machine learning model is trained on the second computer equipment based on the aforementioned partial data points and the output labels received from the first computer equipment.

8. The method as described in claim 7, characterized in that, A subset of the plurality of fields includes an audio field, the value of which includes audio data of a person’s speech, and another subset of the plurality of fields includes a video field, the value of which includes video data of the person’s lips or face while speaking the speech; The first machine learning model is configured to perform speech-to-text conversion based on the complete data points, and the second machine learning model is configured to perform speech-to-text conversion based on the partial data points, wherein the output label includes the text; and wherein the partial feature vector includes the audio field but does not include the video field.

9. The method as described in claim 7, characterized in that, A subset of the plurality of fields includes an image field, the value of which includes image data, and another subset of the plurality of fields includes an inertial sensor data field, the value of which includes inertial sensor data from one or more sensors that measure the motion of the camera when the image data is captured; The first machine learning model is configured to perform image recognition to detect objects based on the complete data points, and the second machine learning model is configured to perform image recognition based on the partial data points, wherein the output label includes an indication of the object; and wherein the portion of the feature vector includes the image field but does not include the inertial sensor data field.

10. The method according to any one of claims 7 to 9, characterized in that, This includes the initial training phase performed by the second computer equipment before receiving the partial data points: Receive multiple initial data points from the first computer equipment, each initial data point including a set of some or all of the corresponding values ​​in the fields of the feature vector; In response, associated labels are generated by inputting the initial data points into the second machine learning model, and the associated labels are sent back to the first computer equipment, thereby enabling the first computer equipment to train the first machine learning model based on the initial data points and the associated labels sent from the second computer equipment.

11. The method according to any one of claims 7 to 9, characterized in that, This includes subsequent stages following training of the second machine learning model by the second computer equipment based on the aforementioned partial data points and associated labels: The second machine learning model is used to generate new labels for new data points that include some or all of the values ​​in the field of the feature vector, and the new labels are sent to the first computer equipment or other computer equipment to train the first machine learning model on the first computer equipment or another machine learning model on the other computer equipment.

12. A second computer apparatus configured to perform the method of any one of claims 7 to 11.

13. A distributed arrangement of computer equipment, comprising the first computer equipment as described in claim 6 and the second computer equipment as described in claim 12.

Images