Federated contrastive training of neural networks with high scalability to large amounts of training data.

Abstract

Method (100) for generating a training contribution for a neural network (1) on a client node C1, ..., CN for federated training of the neural network (1), comprising the steps: • a complete set of parameters W that characterize the behavior of the neural network (1) is received from a server node Q (110); • Records r1, ...,r k Measurement data acquired with various sensors (21-24) belonging to the client node C1, ..., CN are processed by the neural network (1) parameterized in this way, and / or any part of this network (1), into representations z1, ..., z K transformed in a latent space Z (120); • using a given similarity metric (3), similarities S ij of couples (e.g. i , z j ), i,j ∈ {1, ...,K} of representations z1, ..., z K determined (130); • Using a given cost function (4), it is evaluated (140) to what extent these similarities S ij in accordance with a commonality measure Ŝ ij the associated records r i , r j stand; • A proposed change ΔW for the parameters W is optimized to the goal (150) of improving the evaluation (4a) by the cost function (4); • The proposed change ΔW is transmitted to a server node Q (160).

Description

[0001] The present invention relates to a training of neural networks that is scalable to large amounts of training data with high variability. State of the art

[0002] Foundation models serve as the basis for many particularly demanding applications of neural networks, such as language models for text generation. These are neural networks that exhibit a very high ability to generalize to unseen data during training. This ability to generalize is often accompanied by a large number of parameters that characterize the behavior of the neural network. Foundation models can typically have 500 million parameters or more.

[0003] The large number of parameters means that a very large number of training examples are required. This high demand cannot be met for most applications with "labeled" training examples, for which the desired output to be provided by the neural network is known, because labeling often requires manual work and is therefore very expensive. Consequently, foundation models are frequently trained predominantly or entirely on unlabeled training examples. Disclosure of the invention

[0004] The invention provides a method for generating a training contribution for a neural network on a client node C1, ..., CN for federated training of the neural network. A multitude of such training contributions can be combined to form a complete training of the neural network.

[0005] As part of the process, a complete set of parameters W, which characterize the behavior of the neural network, is received by a server node Q. This server node Q orchestrates the federated training and is specifically responsible for collecting the training contributions from all client nodes C1, ..., CN.

[0006] Using the neural network in the state characterized by the parameters W, and / or any part of this network, records r1, ...,r are created. k of measurement data acquired with various sensors belonging to the client node C1, ..., CN, in representations z1, ...,z K transformed in a latent space Z. These representations z1, ..., z K In particular, they may, for example, have a significantly lower dimensionality than the original records r1, ..., r Kof measurement data. That is, they can be characterized by a smaller number of independent numerical values. All representations z1, ...,z K belong to one and the same latent space Z. The dimensionality of the representations z1, ...,z is thus determined. K , that is, the number of independent values ​​of which each representation z1, ..., z K This depends on the architecture of the neural network. The training then ensures that a manifold of representations z1, ...,z exists in the vector space defined by this architecture. K is defined as a latent space Z. This latent space Z thus serves, among other things, to combine measurement data from different measurement modalities.

[0007] Here, the term "record" refers to any data structure that arises in connection with a measurement performed by a sensor. A record is thus comparable, for example, to an index card in a card file containing a collection of information about a specific object or event. The German term "Datensatz" (data set) would be appropriate in principle, but in the field of machine learning, it refers to a "set of records, and especially of training examples," which corresponds to a card file filled with many index cards.

[0008] Using a given similarity metric, similarities S ij of couples (e.g. i , z j ), i,j ∈ {1, ...,K} of representations z1, ...,z KThe similarity can be determined using, for example, the cosine similarity, but also any other suitable metric. For example, a distance measure in space Z can be used, where the similarity S ij The greater the distance, the lower the cost. A predefined cost function is then used to evaluate the extent to which these similarities S ij in accordance with a commonality measure Ŝ ij the associated records r i , r j The more this is the case, the better the evaluation by the cost function. The commonality measure Ŝ ij provides at least a qualitative, and especially a quantitative, statement regarding the extent to which the records r i , r jto refer to identical or similar physical content of the observed scene. In the simplest case, based on prior knowledge, for example regarding the overlap or non-overlap of the detection ranges of sensors, it can be specified whether the records r i , r j have perfect commonality. For example, the scale of the commonality measure Ŝ ij be chosen so that perfect commonality corresponds to the extreme value Ŝ ij = 0 corresponds to vanishing distance, while dissimilarity is expressed in arbitrarily large absolute values ​​of Ŝ ij It may be coded. That the similarities S ij "in accordance" with the commonality measure Ŝ ij Therefore, standing does not necessarily mean that the numerical values ​​of S ij and Ŝ ij numerically of the same order of magnitude and therefore directly comparable. Rather, it is sufficient if S ij and Ŝ ijare correlated in some way.

[0009] A proposed change ΔW for the parameters W is optimized to improve the evaluation by the cost function. This proposed change ΔW is transmitted to a server node Q.

[0010] This training aims to ensure that the representations z in the latent space Z are i , z j of similar records r i , r j move closer together, while representations z i ,z j of dissimilar records r i , r j They move away from each other. This is therefore a contrastive learning process that does not involve labeling the records r1, ...,r. K It works with measurement data and target outputs. The only type of prior knowledge that serves as feedback for the training lies in the commonality measures. ij from Records r i , r jfrom measurement data recorded with various sensors.

[0011] Into these commonalities Ŝ ij This can, in particular, address the extent to which overlapping parts of the same scene are observed by the different sensors. For example, if both a camera and a lidar sensor on a vehicle are observing forward in the direction of travel, it is to be expected that the corresponding records will r i , r jThe measurement data from the two sensors are comparatively similar. Firstly, the spatial detection ranges of the two sensors overlap, meaning that the measurement data at least partially relate to the same objects. Secondly, the measurement modalities are also similar, at least in that at least partially similar features of the scene and the objects within it can be physically captured. For example, both the image and the lidar data depend on the shape of objects, and only those objects in the direct line of sight to the sensor are relevant. Any prior knowledge of this or a related kind can be incorporated into the commonality measures Ŝ ij This will then reveal, in particular, what specific similarities S ij as large as possible and what specific similarities S ij They should be as low as possible.

[0012] In particular, only one type of prior knowledge is required, which is specific to the configuration of the sensors belonging to the respective client node C1, ..., CN. This means that the amount of prior knowledge to be acquired does not depend on the number of records r1, ...,r used for training. K depends on measurement data. This means the training scales very well to arbitrarily large numbers of training examples per client node C1, ..., CN.

[0013] Regarding this scaling, there is also a synergistic effect between contrastive learning on the one hand and federated learning on the other. Federated learning distributes both the task of generating a large number of training examples and the task of processing these training examples across many client nodes C1, ..., CN. In particular, the participation of many client nodes C1, ..., CN also increases the desired high variability in the total set of training examples used, because different client nodes C1, ..., CN have the opportunity to observe different scenarios. The client nodes C1, ..., CN can, for example, belong to vehicles, robots, or other autonomous agents located in different places.

[0014] The hardware requirements and energy consumption for training are also distributed accordingly. Training a Foundation model can easily require 500 graphics cards (GPUs) or more. 300 watts for operating a single GPU can still be drawn from a vehicle's electrical system, while 150 kW for operating 500 GPUs is several times the power consumption typically allocated to a single-family or two-family home.

[0015] Finally, federated training also eliminates data privacy concerns when collecting large amounts of measurement data. Many types of measurement data, such as image and audio data, can be personally identifiable, meaning their collection and sharing are subject to specific legal requirements. This is especially true when data is shared across geopolitical boundaries. However, within the framework of federated training, only proposed changes ΔW are shared, from which the original records r1, ..., r are derived. K The measurement data can no longer be reconstructed. The records r1, ..., r K The measurement data itself remains on the respective client nodes C1, ..., CN.

[0016] Thus, federated learning makes it easier to process very large amounts of training examples, while at the same time contrastive learning makes it possible to manage with unlabeled training examples and prior knowledge that is independent of the number of training examples.

[0017] In a particularly advantageous embodiment, the neural network comprises multiple encoders. It can, for example, be and / or include the union of these encoders. The measurement data from each sensor is fed to an encoder assigned to that sensor. In this way, each encoder can specialize in extracting data from the records r1, ..., r acquired by a specific sensor. K The recorded measurement data represent those pieces of information z1, ..., z Kto adopt features that are important in relation to comparison with other measurement modalities. In particular, the various encoder measurement data, which exist in very different dimensionalities, can be merged into a space Z of representations with a common dimensionality.

[0018] In this context, the behavior of individual encoders can be characterized, for example, by non-overlapping components of the parameters W. Improving the performance of one encoder then does not come at the expense of the performance of another. In most applications, the components of the parameters W are non-overlapping.

[0019] The commonality measures Ŝ ij from Records r i , r j In particular, at least similarities in the semantic content of the records r can be observed. i , r jdepend. As explained previously, this commonality can usually be estimated based on the sensor configuration of each client node C1, ..., CN.

[0020] In particular, for example, the commonality measures Ŝ ij from Records r i , r j The more similar the perspectives are from which the sensors that make these records are rated, the higher the rating will be. i , r j deliver, observe a scene. For example, if a 360-degree lidar scans the entire surroundings of a vehicle and the vehicle also has cameras that look forward or backward in relation to the direction of travel, then the portion of the lidar scan that covers the area ahead in the direction of travel is comparatively similar to the image from the forward-facing camera, but dissimilar to the image from the rear-facing camera.

[0021] Alternatively or in combination with this, the commonality measures Ŝ ij from Records r i , r j The more similar the physical mechanisms of action of the sensors that record these records are, the higher the rating will be. i , r j They are, as previously explained, relatively similar to radar data, since both measurement modalities use visible light and can only detect objects in a direct line of sight. However, camera images are significantly less similar to radar data, as radar beams are preferentially reflected by different types of materials than laser beams and, unlike laser beams, can also pass through certain materials.

[0022] Therefore, the sensors advantageously include at least one camera, and / or at least one radar sensor, and / or at least one lidar sensor.

[0023] The purpose of the training contributions is that by combining many training contributions on one or more server nodes Q, an overall training result for the neural network is obtained. Therefore, the invention also provides a method for federated training of a neural network from the perspective of one or more server nodes Q.

[0024] In this process, a server node Q initializes a complete set of parameters W that characterize the behavior of the neural network. The server node Q then distributes this complete set of parameters W to a plurality of client nodes C1, ..., CN. As previously explained, these client nodes C1, ..., CN can be equipped with many different sensor configurations.

[0025] From each client node C1, ..., CN, change proposals ΔW for the parameters W are determined using the previously described procedure for generating a training contribution and sent to the server node Q. The server node Q receives these change proposals ΔW and updates the set of parameters W based on them. The change proposals ΔW can be aggregated in any way, such as by averaging.

[0026] Afterwards, the set of parameters W can, for example, be redistributed to the client nodes C1, ..., CN. This allows all client nodes C1, ..., CN to be brought back to the same state, and from this, the next proposed changes ΔW can be determined.

[0027] In another particularly advantageous configuration, pairs (t i , r i ) of texts t i on the one hand and Records r iOn the other hand, measurement data is sampled from an already existing, i.e., previously recorded, training dataset. These texts t i and the records r i Measurement data are each processed by the neural network into representations. i or zr i in the latent space Z. Here, the neural network is in a training state in which many training contributions from client nodes C1, ..., CN have already been combined. The records r i The measurement data can relate to one or more sensor modalities. The neural network can then, in particular, for example... • another encoder that translates the texts t i transformed into the latent space Z, and / or • another encoder specifically for the records r ifrom the training dataset transformed into the latent space Z, including, for example, these additional encoders can run on the central server node Q or at least be managed by this server node Q. Alternatively or in combination with this, the records r i They can also be processed, for example, by an encoder trained for the respective measurement modality as described above; that is, this encoder can be reused. The latter preserves the semantic connection between the texts. i on the one hand, and the records r i on the other hand.

[0028] Using a predefined similarity metric for the pairs (t i , r i ) similarities S i the associated representations zt i and zr iThe similarity can be determined using, for example, the cosine similarity, but also any other suitable metric. For example, a distance measure in space Z can be used, where the similarity S i The greater the distance, the lower the cost. A predefined cost function is used to evaluate the extent to which these similarities S i in accordance with a commonality measure derived from the training dataset Ŝ i stand. Here, this measure of commonality can be used. i The training dataset can be stored in any way. For example, the training dataset can consist exclusively of pairs (t i , r i ) of similar texts t i and Records r i of measurement data. However, the training dataset can also contain, for example, positive pairs (t i , r i ) of similar texts t i and Records r iof measurement data (such as a picture of a car in conjunction with the term "car" or "vehicle") as well as negative pairs (t i , r i ) of dissimilar texts t i and Records r i of measurement data (such as a picture of a horse in connection with the term "car" or "vehicle"). It can then, for example, be associated with the respective pair (t i , r i It must be specified whether it is a positive or negative pair. Intermediate levels are also possible.

[0029] For example, if starting from pairs (t i , r i ) of related texts t i and Records r i each representation of the texts t i and Records r i All pairs (zt) can be generated in the latent space Z. i , zr j ) of representations zt i and zr jwith i = j as positive pairs and all pairs with i ≠ j as negative pairs.

[0030] The parameters W are optimized to improve the evaluation by the cost function. This means that the parameters already optimized through federated training are further refined to learn an additional connection between measurement data and text. This allows users of the neural network to interact with it by entering text. For example, the user can input vaguely defined states, such as "flame burns unevenly," or vaguely defined values, such as "not very warm" or "quite cold." Optionally, parameters W relating to encoders for records r can be used. i refer to, be kept constant. Then only the encoder for texts t iTraining continues while one or more encoders are used for records. i to remain at a training level achieved "in the field".

[0031] In one exemplary implementation, the texts t i , and / or the records r i of measurement data, fed to separate encoders that were not used in the generation of training contributions for processing measurement data from individual sensors. Further training for the association of texts t i on the one hand and Records r i The difference between measurement data and other data is then primarily manifested in these separate encoders. However, if appropriate, the parameters of the encoders already trained in federated training can also be further optimized. As explained previously, the latter preserves a previously learned semantic connection between the texts. i and Records r i .

[0032] In a particularly advantageous configuration, the texts include t i Statements and / or claims relating to the Records r i of measurement data. The commonality measures Ŝ i the texts t i to the records r i The more accurate the respective statements and / or claims are, the higher their scores will be. In this way, the neural network can learn to later determine, for records of measurement data not seen during training, whether the statement and / or claim applies to that record.

[0033] For example, a training dataset might contain images of people associated with the statements "person is wearing a jacket" and "person is not wearing a jacket." Pairs of images and statements that match each other, i.e., • Images of people wearing jackets in association with the statement "person wearing a jacket" and • Images of people without jackets associated with the statement "person is not wearing a jacket", These are positive examples with a high degree of commonality. i In contrast, pairs of images and statements that do not match each other are... • Images of people wearing jackets in association with the statement "person is not wearing a jacket" and • Images of people without jackets associated with the statement "person wearing a jacket",

[0034] Negative examples with a low degree of commonality Ŝ i .

[0035] For example, • a first image showing a person wearing a jacket, • a second image showing a person without a jacket, and The statement "person wears jacket" is transformed into representations in the latent space Z. If, according to an arbitrary metric, the representation of the first image is closer to the representation of the statement "person wears jacket" than the representation of the second image, the training is successful, i.e., the cost function applied during training has been successfully minimized. Here, the latent space Z as such has no physical meaning, and absolute distances within it also have no physical meaning. Only a relative comparison of distances provides interpretable insights.

[0036] Similarly, in quality control in manufacturing, images of workpieces to be inspected can be stored in the training data set in association with the statements "workpiece is OK" and "workpiece is not OK".

[0037] The representation learned by the neural network makes it possible to decide for future images (or other records of measurement data) whether the person is wearing a jacket or whether the workpiece is okay. If the record r of measurement data and the text t containing the statement are each translated by the neural network into representations zr and zt, respectively, then the representation zt of the correct statement in the latent space Z is closer to the representation zr of the record r than the representation zt of the incorrect statement.

[0038] In this way, the trained neural network can be used directly to make statements about records r of measurement data. The neural network can also serve as the basis for further processing by a task network trained to solve a predefined task. Through large-scale contrastive and federated learning, the neural network can learn fundamental skills for analyzing measurement data, enabling the creation of representations z in the latent space Z as a generic precursor for a multitude of tasks. A corresponding task network therefore no longer needs to start from scratch but can build upon the representations z in space Z, significantly simplifying the training process.This is somewhat analogous to a learner driver's foundation network, which has already learned to recognize traffic-relevant objects such as other vehicles, traffic signs, or lane markings. Their task network, which still needs training, "only" needs to learn to react correctly to these objects and operate the vehicle accordingly. The same foundation network is used when, as part of their vocational training, learner drivers can already recognize all the necessary tools and equipment and simply need an additional task network to learn how to use them.

[0039] Therefore, in a further particularly advantageous embodiment, records r of measurement data and / or texts t are fed to the trained neural network. The representations zr and zt, respectively, which the neural network has generated from the records r of measurement data and from the texts t, are fed to a task network, which then determines an output with regard to a predefined task. This task can, for example, consist of assigning classification scores to the representations zr and zt with respect to one or more classes of a predefined classification. Thus, the combination of the trained neural network and the task network can, for example, form an image classifier.

[0040] In a further advantageous embodiment, a control signal is derived from the output of the task network. A vehicle, a driver assistance system, a robot, a quality control system, a system for monitoring areas, and / or a medical imaging system is controlled by this signal. This increases the probability that the response executed by the respective technical system to the control signal is appropriate to the situation represented by the records r of measurement data and / or texts t.

[0041] The methods can be wholly or partially computer-implemented. Therefore, the invention also relates to a computer program with machine-readable instructions which, when executed on one or more computers and / or compute instances, cause the computer(s) and / or compute instance(s) to execute one of the described methods. In this sense, vehicle control units and embedded systems for technical devices, which are also capable of executing machine-readable instructions, are also to be considered computers. Compute instances can be, for example, virtual machines, containers, or serverless execution environments, which can be provided, in particular, in a cloud.

[0042] The invention also relates to a machine-readable data carrier and / or a downloadable product containing the computer program. A downloadable product is a digital product that can be transmitted over a data network, i.e., downloaded by a user of the data network, and which can, for example, be offered for immediate download in an online shop.

[0043] Furthermore, one or more computers and / or compute instances can be equipped with the computer program, the machine-readable data carrier, or the download product.

[0044] Further measures improving the invention are described in more detail below, together with a description of preferred embodiments of the invention, with reference to figures. Examples of implementation

[0045] It shows: Fig. 1 Exemplary embodiment of method 100 for generating a training contribution for a neural network 1 on a client node C1, ..., CN; Fig. 2. Illustration of the contrastive optimization goal in procedure 100; Fig. 3 Exemplary implementation of method 200 for federated training of a neural network 1; Fig. 4 Illustration of the contrastive optimization goal in procedure 200; Fig. 5 Exemplary arrangement of server node Q and client node C1, ..., CN for the implementation of procedures 100 and 200.

[0046] Fig. Figure 1 is a schematic flowchart of an embodiment of method 100 for generating a training contribution for a neural network 1 on a client node C1, ..., CN.

[0047] In step 110, a complete set of parameters W, which characterize the behavior of neural network 1, is received from a server node Q.

[0048] In step 120, records r1, ..., r are... K of measurement data acquired with various sensors 21-24 belonging to the client node C1, ..., CN, with the neural network 1 parameterized according to the parameters W, and / or any part of this network 1, in representations z1, ...,z K transformed in a latent space Z.

[0049] According to Block 121, the neural network 1 can comprise multiple encoders 11-14. According to Block 122, the measurement data from each sensor 21-24 can then be fed to an encoder 11-14 assigned to that sensor 21-24. In particular, according to Block 122a, the behavior of each individual encoder 11-14 can then be characterized by non-overlapping components of the parameter W.

[0050] In step 130, 3 similarities are calculated using a predefined similarity metric. ij of couples (e.g. i , z j ), i, j ∈ {1, ..., K} of representations z1, ..., z K determined.

[0051] In step 140, a given cost function 4 is used to evaluate the extent to which these similarities S ij in accordance with common measures Ŝ ij the associated records r i , r j Standing. A rating of 4a is generated.

[0052] According to Block 141, the commonality measures Ŝ ij from Records r i , r j at least similarities in the semantic content of the records r i , r j depend.

[0053] According to Block 142, the commonality measures Ŝ ij from Records r i , r j the higher they will be rated • the more similar the perspectives are from which sensors 21-24, which produced these records, i , r j deliver, observe a scene, and / or • the more similar the physical mechanisms of action of sensors 21-24 that produce these records r i , r j deliver, are.

[0054] In step 150, a proposed change ΔW for the parameters W is optimized to improve the evaluation 4a using the cost function 4. This means that candidates for this proposed change ΔW are identified and tested by applying them in step 120 until a predefined termination criterion is met. Depending on the specific design, various optimization methods can be used for this step, such as stochastic gradient descent and its derivatives. The termination criterion could, for example, be a specific value for the evaluation 4a, a decreasing rate of change for this evaluation 4a, and / or a maximum total number of iterations.

[0055] In step 160, the final change proposal ΔW is transmitted to a server node Q.

[0056] The optimization goal pursued within the framework of procedure 100 is in Fig. Figure 2 illustrates this in more detail. In this simple example, three cameras mounted on a vehicle, acting as sensors 21-23, deliver images as records r1, r2, r3 of measurement data. Cameras 21 and 22 are positioned at an angle to each other and observe an area ahead in the direction of travel from two perspectives. The third camera, 23, looks to the side from the direction of travel.

[0057] In the Fig. In the depicted state 2, cameras 21 and 22 observe one and the same vehicle from different perspectives. Since it is the same vehicle, the commonality measure Ŝ 12 The records (images) r1 and r2 are large. If neural network 1 is "well trained", these records r1 and r2 will be mapped to representations z1 and z2 that are close to each other and thus have a high degree of similarity S. 12 to each other.

[0058] The third camera, 23, which looks to the side, sees something completely different, in this example a dog. The commonality measure Ŝ 23 The similarity between record r3 and record r2 (or even record r1, which is omitted for clarity) is low. Therefore, record r3 should be mapped to a representation z3 that is far removed from representation z2, i.e., has only a low similarity S. 23 to this representation z2 has.

[0059] Fig. Figure 3 is a schematic flowchart of an embodiment of method 200 for federated training of a neural network 1. This method is shown from the perspective of a server node Q.

[0060] In step 210, a server node Q initializes a complete set of parameters W that characterize the behavior of the neural network (1).

[0061] In step 220, the complete set of parameters W is distributed from the server node Q to a plurality of client nodes C1, ..., CN.

[0062] In step 230, the client nodes C1, ..., CN determine 100 change proposals ΔW for the parameters W as part of the procedure and send them to the server node Q.

[0063] In step 240, the set of parameters W is updated by the server node Q based on the proposed changes ΔW.

[0064] In step 250, the updated set of parameters W is redistributed to the client nodes C1, ..., CN. This can be repeated until a predefined termination criterion, such as a maximum total number of iterations, is met.

[0065] In step 260, pairs (t i , r i ) of texts t i on the one hand and Records r i On the other hand, measurement data was sampled from a training dataset 5.

[0066] In step 270, the texts t i and the records r i of measurement data each with the neural network 1 in representations zt i or zr i transformed in the latent space Z.

[0067] According to Block 271, the texts t i , and / or the records r i of measurement data, separate encoders 15-16, which were not used in the context of generating training contributions for processing measurement data from individual sensors 21-24.

[0068] In step 280, a predefined similarity metric 6 is used for the pairs (t i , r i ) each similarities S i the associated representations zt i and zr i determined.

[0069] In step 290, a predefined cost function 7 is used to evaluate the extent to which these similarities S iin accordance with a commonality measure derived from the training dataset Ŝ i Standing. A rating of 7a is generated.

[0070] According to Block 261, the texts t i Statements and / or claims relating to the Records r i of measurement data. According to Block 291, the commonality measure Ŝ can then be i the texts t i to the records r i The more accurate the respective statements and / or claims are, the higher the rating will be.

[0071] In step 300, the parameters W are optimized to improve the evaluation 7a using the cost function 7. The fully optimized state of the parameters W is denoted by the reference symbol W* and defines the fully trained state 1* of the neural network 1. This can also be distributed to the client nodes C1, ..., CN.

[0072] In step 310, records r of measurement data and / or texts t are fed into the trained neural network 1*. Accordingly, neural network 1* forms representations zr and zt, respectively, in the latent space Z.

[0073] These representations zr and zt are fed into a task network 8 in step 320, whereupon this task network 8 determines an output 9 with respect to a given task.

[0074] In step 330, a control signal 330a is determined from the output 9 of the task network 8.

[0075] In step 340, a vehicle 50, a driver assistance system 51, a robot 60, a quality control system 70, a monitoring system 80, and / or a medical imaging system 90 is controlled with the control signal 330a.

[0076] Fig. Figure 4 illustrates the optimization goal pursued in the optimization process of Procedure 200. In this simple example, there is only one text t1 and two images as records r1 and r2 of measurement data. The text t1 reads "OK" and, in a quality control application, indicates that a tested workpiece is OK. The two records (images) r1 and r2 show two workpieces, in this case, nuts and bolts.

[0077] Record r1 depicts a nut that is perfectly fine. Accordingly, the similarity measure Ŝ(t1,r1) of this record r1 to the text t1 is high. Consequently, neural network 1 ideally maps record r1 to a representation zr1 that is close to the representation zt1 of the text t1.

[0078] Record r2, on the other hand, shows a nut with a defect F, in this case a through crack. Accordingly, the commonality measure Ŝ(t1,r2) of this record r2 to the text t1 is low. Consequently, the neural network 1 ideally maps record r2 to a representation zr2 that is further removed from the representation zt1 of the text t1. However, it is still close to representation zr1, since it is still an image of a nut, just one with the defect F.

[0079] In Fig. Figure 5 depicts a complete system consisting of a server node Q and several client nodes C1, ..., CN for the federated training of a neural network 1. In this example, each client node C1, ..., CN has two cameras 21 and 22, a lidar sensor 23, and a radar sensor 24. These sensors 21-24 provide records r1-r4 of measurement data. Each sensor 21-24 is assigned its own encoder 11-14 of the neural network 1, which receives the respective records r1-r4 of measurement data and processes them into representations z1-z4. As described previously, actual pairwise similarities S are then generated. ij of representations z i and z j with a cost function 4, evaluated to what extent it corresponds to commonality measures Ŝ ij the associated records r i , r jare in agreement. This results in ratings 4a. On each client node C1, ..., CN, the optimization of neural network 1 for the best possible rating 4a results in a proposed change ΔW for the parameters W of neural network 1. This proposed change ΔW is transmitted to the server node Q.

[0080] In step 240 of procedure 200, server node Q combines the proposed changes ΔW to create a new version of the parameters W. This is then used in the Fig. The example shown in point 5 has been refined once more. From a training dataset 5, pairs (t) are created. i , r i ) of texts t i on the one hand and Records r i on the other hand, sampled from measurement data. The texts t i then become representations zt i and the records r i from measurement data to representations zr iIt is then processed. A cost function 7 is used to evaluate the extent to which actual similarities S i the representations zt i and zr i with a commonality measure S derived from training data set 5 i stand. A rating of 7a is generated. In which in Fig. In the example shown in section 5, the optimization for the best possible rating (7a) leads to the version of the parameter W that is redistributed to the client nodes C1, ..., CN. However, it is also possible, for example, to first complete the federated training in cooperation with the client nodes C1, ..., CN and then, on the server node Q, to include the texts t i to optimize.

Claims

[1] Method (100) for generating a training contribution for a neural network (1) on a client node C1, ..., CN for a federated training of the neural network (1), comprising the steps: • a complete set of parameters W characterizing the behavior of the neural network (1) is received from a server node Q (110); • Records r1, ...,r k of measurement data recorded with various sensors (21-24) belonging to the client node C1, ..., CN are processed with the thus parameterized neural network (1), and / or any part of this network (1), into representations z1, ..., z K transformed into a latent space Z (120); • with a given similarity metric (3) similarities S ij of couples (e.g. i , e.g. j ), i,j ∈ {1, ...,K} of representations z1, ..., z K determined (130); • with a given cost function (4) it is evaluated (140) to what extent these similarities S ij in accordance with a commonality measure Ŝ ij the associated records r i , r j stand; • a change proposal ΔW for the parameters W is optimized (150) to the goal of improving the evaluation (4a) by the cost function (4); • the change proposal ΔW is transmitted to a server node Q (160). [2] Method (100) according to claim 1, wherein • the neural network (1) comprises several encoders (11-14) (121) and • the measurement data of each sensor (21-24) are fed (122) to an encoder (11-14) assigned to this sensor (21-24). [3] Method (100) according to claim 2, wherein the behavior of the individual encoders (11-14) is each characterized by non-overlapping portions of the parameters W (122a). [4] Method (100) according to one of claims 1 to 3, wherein the commonality measures Ŝ ij von Records r i , r j at least of similarities of the semantic contents of the records r i , r j depend (141). [5] Method (100) according to one of claims 1 to 4, wherein the commonality measures Ŝ ij von Records r i , r j be valued all the more highly (142), • the more similar the perspectives are from which the sensors (21-24) that record these i , r j deliver, observe a scene, and / or • the more similar the physical mechanisms of action of the sensors (21-24) that record these i , r j deliver, are. [6] Method (100) according to one of claims 1 to 5, wherein the sensors (21-24) • at least one camera, and / or • at least one radar sensor, and / or • at least one lidar sensor include. [7] Method (200) for federated training of a neural network (1) comprising the steps: • a server node Q initializes (210) a complete set of parameters W that characterize the behavior of the neural network (1); • the complete set of parameters W is distributed by the server node Q to a plurality of client nodes C1, ... , CN (220); • within the framework of the method (100), the client nodes C1, ..., CN determine change proposals ΔW for the parameters W and send them to the server node Q (230); • based on the proposed changes ΔW, the set of parameters W is updated by the server node Q (240). [8] The method (200) of claim 7, wherein the updated set of parameters W is redistributed (250) to the client nodes C1, ..., CN. [9] Method (200) according to one of claims 7 to 8, wherein • Couples (t i , r i ) of texts t i on the one hand and Records r i of measurement data on the other hand are sampled from a training data set (5) (260); • the texts t i and the Records r i of measurement data with the neural network (1) in representations zt i or zr i be transformed in the latent space Z (270); • with a given similarity metric (6) for the pairs (t i , r i ) similarities S i the corresponding representations zt i and zr i be determined (280); • using a given cost function (7) it is evaluated (290) to what extent these similarities S i in accordance with a commonality measure Ŝ resulting from the training data set (5) i stand; • the parameters W are optimized (300) to improve the evaluation (7a) by the cost function (7). [10] Method (200) according to claim 9, wherein the texts t i , and / or the Records r i of measurement data, are fed to separate encoders (15-16) (271), which were not used in the context of generating training contributions for the processing of measurement data from individual sensors (21-24). [11] Method (200) according to one of claims 9 to 10, wherein • the texts t i Statements and / or allegations regarding the Records r i of measurement data (261) and • the commonality measure Ŝ i the texts t i to the Records r i the more accurate the respective statements and / or claims are, the higher the rating is (291). [12] Method (200) according to one of claims 9 to 11, wherein • records r of measurement data and / or texts t are fed to the trained neural network (1*) (310); and • representations zr or zt formed therefrom with the neural network (1*) in the latent space Z are fed (320) to a task network (8) which determines an output (9) with regard to a given task. [13] Method (200) according to claim 12, wherein • a control signal (330a) is determined (330) from the output (9) of the task network (8); and • a vehicle (50), a driver assistance system (51), a robot (60), a quality control system (70), a system (80) for monitoring areas, and / or a medical imaging system (90), with which the control signal (330a) is controlled (340). [14] A computer program comprising machine-readable instructions which, when executed on one or more computers and / or compute instances, cause the computer(s) and / or compute instances to perform a method (100, 200) according to any one of claims 1 to 13. [15] A machine-readable data carrier containing the computer program according to claim 14. [16] One or more computers and / or compute instances equipped with the computer program according to claim 14 and / or with the machine-readable data carrier according to claim 15.

Images