Training a tokenizer network to support the training of transformer networks on point clouds

Abstract

Method (100) for training a tokenizer network (2) which quantizes vectorial representations of point clouds of measurement data in the space of said representations, said point clouds each assigning measured values of one or more measurement variables to discrete points in the space, comprising the following steps: - training examples (3) of point clouds of measurement data are provided (110); - these training examples (3) are converted (120) into vectorial representations (1a) using an encoder network (1); - these vectorial representations (1a) are quantized (130) using the tokenizer network (2) to be trained; - clusters or other accumulations (4a) of points in the point cloud of the respective training example (3) and / or processing products (4b) of such clusters or accumulations (4a) are determined (140) from the quantized vectorial representations (2a) using a decoder arrangement (4) of one or more further networks (4', 4''); - a predefined cost function (5) is used to measure (150) the extent to which these clusters or accumulations (4a) are consistent with the original point cloud; and - at least parameters (2b) which characterize the behaviour of the tokenizer network (2) are optimized (160) for the aim of improving the assessment (5a) by means of the cost function (5).

Description

[0001] R.415044

[0002] - 1 -

[0003] Description

[0004] Title:

[0005] Training a tokenizer network to support the training of transformer networks on point clouds

[0006] The present invention relates to the training of neural transformer networks on point clouds, and in particular on sparsely populated and noisy point clouds, such as those encountered in the environmental monitoring of vehicles and robots.

[0007] State of the art

[0008] Neural transformer networks are particularly well-suited for processing sequences of input. They are therefore used for speech processing, but also, for example, for a variety of tasks in the field of computer vision, such as image classification or semantic segmentation.

[0009] The inputs received by a transformer network are typically converted into vector representations, which can be understood as a sequence of so-called "tokens" with one or more vector components. When training transformer networks, individual "tokens" in a training example are often "masked," meaning they are ignored or obscured. The transformer network is then trained to predict these "masked" tokens.

[0010] When processing speech as input, it is not particularly difficult to check whether the "masked" tokens are included in the output of the Transformer network, due to the overall limited vocabulary in this context. However, if the input includes, for example, images, then R.415044

[0011] - 2 - It is common practice to quantize the vectorial representation used for comparison with the output of the transformer network into a restricted space of possibilities using a so-called tokenizer network. Such a tokenizer network, in turn, requires training.

[0012] Disclosure of the invention

[0013] The invention provides a method for training a tokenizer network. This tokenizer network quantizes vectorial representations of point clouds of measurement data in the space of these representations. The point clouds assign measured values ​​of one or more quantities to discrete points in space. For example, in radar or lidar reflections, each point in space can be annotated with the intensity of a reflected beam, the direction of this reflected beam, and other quantities of interest.

[0014] As part of the process, training examples of point clouds from measurement data are provided. These training examples are converted into vector representations (also called "embeddings") using an encoder network. These vector representations can, for example, have a lower dimensionality than the training examples. "Lower dimensionality" in this context can mean, for example, that the vector representations depend on fewer variables than the original point clouds.

[0015] The vector representations (“embeddings”) are quantized using the tokenizer network to be trained. From the resulting quantized vector representations, clusters or other accumulations of points in the point cloud of the respective training example, and / or processing products of such clusters or accumulations, are determined using a decoder array consisting of one or more additional networks.

[0016] A predefined cost function is then used to determine the extent to which these clusters or accumulations correspond to the original point cloud. This may also be evident from processed products obtained from the clusters or accumulations. R.415044

[0017] - 3 -

[0018] At least the parameters that characterize the behavior of the tokenizer network are optimized with the goal of improving the evaluation by the cost function.

[0019] It was recognized that this method, particularly when used with point clouds generated during the environmental monitoring of vehicles and / or robots, allows for unsupervised training without manual labeling of the training examples. This type of environmental monitoring typically requires covering large spatial areas to capture a complete scene (such as a traffic situation). At the same time, there is usually relative movement between the sensor used to capture the point clouds and the observed scene. This relative movement causes significant noise in the point cloud measurement data. Furthermore, the need to capture a complete scene results in a point cloud containing many objects while simultaneously being relatively sparsely populated.Therefore, individual objects in these point clouds are comparatively difficult to identify. Testing whether the decoder array detects plausible clusters or other accumulations is thus a significantly better indicator of whether the tokenizer network being trained is working well, especially for point clouds of this origin.

[0020] The improved training capabilities of the tokenizer network, in turn, improve the training of a transformer network. Consequently, the transformer network can then perform better on point clouds generated during environmental monitoring of vehicles or robots. This improves the effectiveness of this environmental monitoring. Therefore, the tokenizer network training proposed here significantly contributes to the improved navigation of the vehicle or robot within a given environment.

[0021] In a particularly advantageous embodiment, parameters that characterize the behavior of the encoder network and / or parameters that characterize the behavior of the decoder arrangement are also applied to target R.415044.

[0022] - 4 - optimized to improve the evaluation by the cost function. In this way, not only can the performance of the encoder network or decoder array be improved with regard to the detection of clusters or other accumulations, but further learning progress of the tokenizer network, which is the primary target for training, can also be stimulated. The combination of the trained encoder network and the trained tokenizer network can be further used in the subsequent training of a transformer network to provide the transformer network with suitable training examples. This will be discussed later.

[0023] In a further, particularly advantageous embodiment, the point cloud of the training example is examined for clusters or accumulations using a predefined alternative method. The extent to which these clusters or accumulations correspond to those identified by the decoder array is then measured. As previously explained, this type of measurement captures precisely those features that are particularly easy to identify in point clouds from the environmental monitoring of vehicles or robots.

[0024] For example, the given alternative method can encompass k-means clustering into a predefined number k of clusters. This predefined number k of clusters can then be used, for example, as a hyperparameter that is also optimized during training.

[0025] In a further particularly advantageous embodiment, bounding boxes are determined from the clusters or aggregations identified by the alternative method, and these bounding boxes describe the clusters or aggregations. Simultaneously, the decoder arrangement is configured to recognize objects that correspond to clusters or aggregations of points. The decoder arrangement can, for example, be configured to also specify bounding boxes for the recognized objects. These bounding boxes can then be particularly well compared with the bounding boxes that correspond to the clusters or aggregations identified by the alternative method, such as the "mean intersection over union" (mloU. R.415044).

[0026] - 5 -

[0027] In a further particularly advantageous embodiment, the encoder network comprises one or more PointNet architectures. In each such PointNet architecture, all points of an input point cloud are processed by at least one common multi-layered perceptron. In this way, a PointNet architecture is suitable for extracting both local and global features from a point cloud with arbitrary orientation.

[0028] As previously explained, in a further, particularly advantageous implementation, training examples are selected using point clouds whose points represent locations from which a radar or lidar interrogation beam has been reflected. In practical measurement operations, this type of point cloud tends to be sparsely populated and exhibits clusters or other accumulations as its main features. The training methods typically used for training tokenizers struggle with this type of training data. The method proposed here is specifically designed to utilize precisely this type of training data and to prepare the tokenizer network for subsequent operation with data of this structure.

[0029] This is particularly true in a further, especially advantageous embodiment, in which training examples are selected using point clouds whose measurement data were acquired by at least one sensor mounted on a vehicle and / or robot. As explained previously, this type of point cloud exhibits a particularly high level of noise due to the relative motion between the sensor and the observed scene. Furthermore, this application requires monitoring a particularly large spatial area to ensure timely responses to all objects and / or events relevant to the future trajectory planning of the vehicle or robot.

[0030] In a further particularly advantageous embodiment, additional training examples of point clouds from measurement data are provided. These training examples are converted into vector representations using the trained encoder network. These vector representations are then quantized using the trained tokenizer network. R.415044

[0031] - 6 -

[0032] Simplified versions of the vector representations are generated by masking one or more components. In this context, masking refers to any replacement with a placeholder value that obscures the original value of the component but still indicates that a value is missing at the corresponding position. The simplified version of the vector representation then acts as a kind of "numerical fill-in-the-blank exercise" that the transformer network being trained is intended to complete.

[0033] For this purpose, the thinned-out versions are fed into a transformer network to be trained. A further transformer cost function, which relates to the training of the transformer network, is used to measure the extent to which the output of the transformer network to be trained contains the components masked in the thinned-out versions – in other words, whether the "gaps" in the numerical "gap text" are filled correctly. For this measurement, the quantized, but not thinned-out, vector representations are used. Optionally, the cost function can also measure the extent to which the unmasked components of the thinned vector representation remain unchanged, i.e., whether the transformer network truly only fills the "gaps" in the "gap text" and does not completely rewrite the "text".

[0034] Parameters that characterize the behavior of the transformer network being trained are optimized to improve the evaluation by the subsequent cost function. The fully optimized state of these parameters characterizes the fully trained state of the transformer network. This enables the transformer network to solve tasks on point clouds as input data.

[0035] In particular, the Transformer network can be pre-trained generically using the method proposed here, independent of a specific task. It can therefore be trained unsupervised on a large number of training examples that do not need to be annotated ("labeled") with objects actually present in the respective scenarios or with other prior knowledge ("ground truth"). By "filling in" numerical "gap texts," the Transformer network can thus learn all the basic skills that R.415044

[0036] - 7 - are not specific to a particular task. It can then serve as a "foundation model" that can be adapted to a specific task through subsequent task-specific training. This is somewhat analogous to "foundation models" that are generically pre-trained on large amounts of text or images and can be used for many tasks in the field of natural language processing or computer vision. Especially when using a model for several different tasks, it is sufficient to adapt it to the new task based on the training status of the pre-trained "foundation model." This requires significantly less computational effort and fewer training examples than completely new training from scratch. In particular, fewer task-specific training examples are needed for supervised task-specific training.

[0037] In a further, particularly advantageous configuration, additional training examples of point clouds from measurement data are provided. These additional training examples are labeled with their respective target outputs, which the transformer network being trained should generate from them. The target outputs relate to a specific task that the transformer network being trained is intended to solve.

[0038] The additional labeled training examples are processed by the transformer network being trained into outputs. Deviations of these outputs from the respective target outputs are evaluated using a further predefined task-cost function. Parameters that characterize the behavior of the transformer network being trained are optimized to improve the evaluation by the additional task-cost function.

[0039] Training for a specific task can be combined in any way with unsupervised training using thinned-out versions of vector representations of point clouds. However, as explained earlier, it is also advantageous to first pretrain the transformer network generically and unsupervised (or self-supervised) with thinned-out versions of vector representations of point clouds and then train this generically pretrained transformer network with R.415044 in relation to a specific task.

[0040] - 8 - labeled training examples for this task to be used for further training. A generic pre-training can then be used repeatedly as a starting point for ever new tasks.

[0041] Thus, the method proposed here ultimately makes it possible to apply the concept of generic pretraining and subsequent task-specific training, which has proven successful for language models and image processing models, to transformer networks that process point clouds as input data.

[0042] The transformer network to be trained can, for example, be specifically trained to:

[0043] • to assign classification scores to an entered point cloud or parts thereof with respect to one or more classes of a predefined classification; and / or

[0044] • to semantically segment the input point cloud; and / or

[0045] • To recognize objects in the input point cloud; and / or

[0046] • to identify at least one action based on the entered point cloud that performs at least one entity.

[0047] These are tasks that arise particularly frequently in the environmental monitoring of vehicles and / or robots. At the same time, transformer networks are especially well-suited for solving these tasks—provided they have received sufficient training. The method proposed here ensures this sufficient training.

[0048] In a further particularly advantageous embodiment, point clouds of measurement data acquired with at least one sensor are fed to the trained transformer network. A control signal is derived from the output of the trained transformer network. A vehicle, a driver assistance system, a robot, a quality control system, a system for monitoring areas, and / or a medical imaging system is controlled by this signal. The improved training of the transformer network increases the probability that the response executed by the respective controlled technical device to the control signal will be accurate.

[0049] - 9 -

[0050] The system is appropriate to the respective situation embodied in the point clouds of measurement data.

[0051] In this context, it is particularly advantageous that the Transformer network can be trained more effectively on point clouds, which encompass an entire scene and where individual objects are not necessarily immediately recognizable. Common foundation models that work with point clouds are trained on the three-dimensional reconstruction of individual objects. However, for assessing a traffic situation in which the vehicle or robot is located, evaluating the situation as a whole is crucial. For example, the possible next actions of a road user are often limited by the presence of other road users.

[0052] Therefore, in a further particularly advantageous embodiment, the point clouds of measurement data acquired with the at least one sensor cover a spatial environment in which the vehicle or robot moves, with a dimension of at least 50 meters, preferably at least 100 meters, in at least one spatial direction. This dimension is sufficient to fully capture a traffic situation. If the respective dimension then extends, for example, to a maximum of 200 meters, this is advantageous with regard to the resolution with which details of this traffic situation can be analyzed.

[0053] The method can be wholly or partially computer-implemented. Therefore, the invention also relates to a computer program with machine-readable instructions which, when executed on one or more computers and / or compute instances, cause the computer(s) and / or compute instance(s) to execute the described method. In this sense, vehicle control units and embedded systems for technical devices, which are also capable of executing machine-readable instructions, are also to be considered computers. Compute instances can be, for example, virtual machines, containers, or serverless execution environments, which can be provided, in particular, in a cloud. R.415044

[0054] - 10 -

[0055] The invention also relates to a machine-readable data carrier and / or a downloadable product containing the computer program. A downloadable product is a digital product that can be transmitted over a data network, i.e., downloaded by a user of the data network, and which can, for example, be offered for immediate download in an online shop.

[0056] Furthermore, one or more computers and / or compute instances can be equipped with the computer program, the machine-readable data carrier, or the download product.

[0057] Further measures improving the invention are described in more detail below, together with a description of preferred embodiments of the invention, with reference to figures.

[0058] Examples of implementation

[0059] It shows:

[0060] Figure 1 Exemplary embodiment of method 100 for training a tokenizer network 2;

[0061] Figure 2 Schematic illustration of the processing chain during training.

[0062] Figure 1 is a schematic flowchart of an embodiment of method 100 for training a tokenizer network 2. This tokenizer network 2 quantizes vectorial representations of point clouds of measurement data in the space of these representations. The point clouds assign measured values ​​of one or more measurands to discrete points in space.

[0063] Step 110 provides training examples 3 of point clouds from measurement data. R.415044

[0064] - 11 -

[0065] According to Block 111, in particular, training examples 3 with point clouds can be chosen, the points of which represent locations from which a radar or lidar interrogation beam has been reflected.

[0066] Alternatively or in combination with this, according to Block 112, in particular training examples 3 with point clouds can be selected, the measurement data of which were recorded by at least one sensor carried on a vehicle 50 and / or robot 60.

[0067] In step 120, the training examples 3 are converted into vector representations 1a using an encoder network 1. These vector representations 1a have a lower dimensionality than the training examples 3 in the example shown in Figure 1. However, this is generally not strictly necessary.

[0068] According to Block 121, the encoder network 1 can, for example, comprise one or more PointNet architectures. In each PointNet architecture, all points of an input point cloud are processed by at least one common multilayer perceptron.

[0069] In step 130, the vector representations 1 a are quantized with the tokenizer network 2 to be trained.

[0070] In step 140, using a decoder arrangement 4 consisting of one or more networks 4', 4", clusters or other accumulations 4a of points in the point cloud of the respective training example 3, and / or processing products 4b of such clusters or accumulations 4a, are determined from the quantized vectorial representations.

[0071] In step 150, a predefined cost function 5 is used to measure the extent to which these clusters or accumulations 4a correspond to the original point cloud. This results in an evaluation 5a. R.415044

[0072] - 12 -

[0073] According to Block 151, the point cloud of training example 3 can be examined for clusters or accumulations using a predefined alternative method. It can then be measured, according to Block 152, to what extent these clusters or accumulations correspond to the clusters or accumulations 4a determined by decoder arrangement 4.

[0074] In this context, according to Block 151a, the specified alternative method can, for example, include k-means clusters in a specified number k of clusters.

[0075] According to Block 152a, bounding boxes can be determined from the clusters or aggregations identified using the alternative method, which describe these clusters or aggregations. The decoder arrangement 4 can then, according to Block 152b, be configured to recognize objects that correspond to clusters or aggregations of points.

[0076] In step 160, at least parameter 2b, which characterizes the behavior of the tokenizer network 2, is optimized to improve the evaluation 5a through the cost function 5. This results in

[0077] • a fully optimized state 1 b* of the parameters 1 b, which defines the fully trained state 1* of the encoder network 1;

[0078] • a fully optimized state 2b* of the parameters 2b, which defines the fully trained state 2* of the tokenizer network; and

[0079] • a fully optimized state 4c* of the parameters 4c, which defines the fully trained state 4* of the decoder arrangement 4.

[0080] According to Block 161, additionally parameters 1b, which characterize the behavior of the encoder network 1, and / or parameters 4c, which characterize the behavior of the decoder arrangement 4, can also be optimized to improve the evaluation 5a by the cost function 5.

[0081] In the following, only the trained encoder network 1* and the trained tokenizer network 2* will be used. The trained decoder array 4* has fulfilled its purpose after the tokenizer network 2 has finished training. R.415044

[0082] - 13 -

[0083] In the example shown in Figure 1, from step 170 onwards, a generic, unsupervised or self-supervised training of a Transformer network 7 is carried out.

[0084] In step 170, further training examples 6 of point clouds of measurement data are provided.

[0085] In step 180, these training examples 6 are converted into vector representations 1 a using the trained encoder network 1*.

[0086] In step 190, these vector representations 1 a are quantized using the trained tokenizer network 2*.

[0087] In step 200, thinned versions 1 a' of the vectorial representations 1 a are generated by masking one or more components. These thinned versions 1 a' were not previously quantized by the trained tokenizer network 2*.

[0088] In step 210, these thinned-out versions 1 a' are fed to a transformer network 7, 7* to be trained.

[0089] According to Block 211, in particular, a transformer network 7, 7* can be chosen to be trained, which is trained to

[0090] • to assign classification scores to an entered point cloud or parts thereof with respect to one or more classes of a predefined classification; and / or

[0091] • to semantically segment the input point cloud; and / or

[0092] • To recognize objects in the input point cloud; and / or

[0093] • to identify at least one action based on the entered point cloud that performs at least one entity.

[0094] In step 220, another transformer cost function 8 is used to measure the extent to which the output 7a of the transformer network 7 to be trained contains the components masked in the thinned-out versions 1 a'. For this R.415044

[0095] - 14 -

[0096] The quantized vector representations 2a are used for measurement. This can, in particular, involve, for example, comparing the values ​​predicted by the transformer network 7 for the masked components with the quantized values ​​of the corresponding components in the quantized vector representations 2a. This results in an evaluation 8a.

[0097] In step 230, parameters 7b, which characterize the behavior of the transformer network 7 to be trained, are optimized to improve the evaluation 8a through the further transformer cost function 8. The resulting fully optimized state 7b* of the parameters 7b defines the generically pre-trained state 7* of the transformer network 7.

[0098] In the example shown in Figure 1, from step 240 onwards, the training of the generically pre-trained Transformer Network 7* is carried out for a given task.

[0099] In step 240, further training examples 9 of point clouds of measurement data are provided. These further training examples 9 are labeled with their respective target outputs 10, which the transformer network 7* to be trained should generate from them. Thus, the task-specific training is supervised training.

[0100] In step 250, these further training examples 9 are processed by the transformer network 7* to be trained into outputs 7a.

[0101] In step 260, deviations of these expenditures 7a from the respective target expenditures 10 are evaluated using a further predefined task cost function 11. This results in an evaluation 11a.

[0102] In step 270, parameters 7b, which characterize the behavior of the transformer network 7* to be trained, are optimized to improve the evaluation 11a through the further task cost function 11. The fully optimized state 7b** of the parameters 7b defines the task-specific, fully trained state 7** of the transformer network 7 to be trained. R.415044

[0103] - 15 -

[0104] In the example shown in Figure 1, the concrete use of this fully trained Transformer Network 7** on measurement data continues from step 280 onwards.

[0105] In step 280, the trained transformer network is fed 7** point clouds 3* of measurement data, which were recorded with at least one sensor 12.

[0106] In step 290, a control signal 290a is determined from the output 7a of the trained transformer network 7**.

[0107] In step 300, a vehicle 50, a driver assistance system 51, a robot 60, a quality control system 70, a system 80 for monitoring areas, and / or a medical imaging system 90 is controlled with the control signal 290a.

[0108] According to Block 281, the point clouds 3* can cover a spatial environment in which the vehicle 50 or the robot 60 moves, with an extent of at least 50 meters, preferably at least 100 meters, in at least one spatial direction.

[0109] Figure 2 schematically illustrates the processing of a training example 3 of a point cloud of measurement data during the training of the tokenizer network 2, which can optionally also be combined with training of the encoder network 1 and / or with training of the decoder arrangement 4.

[0110] The point cloud 3 is processed by the encoder network 1 into a vector representation 1a. This vector representation 1a is quantized by the tokenizer network 2. The resulting quantized vector representation 2a is examined by a decoder network 4' for clusters or other accumulations 4a of points. These clusters or accumulations 4a are then examined for objects by an object detector 4" . Thus, the clusters or accumulations 4a are transformed into "pseudo-objects" as processing products. R.415044

[0111] - 16 - duct 4b. These “pseudo-objects” are then checked to see if they are consistent with the original point cloud 3. The decoder network 4' and the object detector 4" together form the decoder arrangement 4.

Claims

R.415044 - 17 - Claims 1. Method (100) for training a tokenizer network (2) that quantizes vectorial representations of point clouds of measurement data in the space of these representations, wherein these point clouds assign measured values ​​of one or more measurands to discrete points in space, comprising the steps: • Training examples (3) of point clouds of measurement data are provided (110); • these training examples (3) are converted into vector representations (1a) using an encoder network (1) (120); • these vectorial representations (1 a) are quantized with the tokenizer network (2) to be trained (130); • from the quantized vectorial representations (2a) clusters or other accumulations (4a) of points in the point cloud of the respective training example (3), and / or processing products (4b) of such clusters or accumulations (4a), are determined (140) using a decoder arrangement (4) from one or more further networks (4', 4"). • It is measured (150) using a predefined cost function (5) to what extent these clusters or accumulations (4a) are consistent with the original point cloud; and • at least parameters (2b) that characterize the behavior of the tokenizer network (2) are optimized to the goal (160) of improving the evaluation (5a) by the cost function (5).

2. Method (100) according to claim 1, wherein additionally parameters (1b) that characterize the behavior of the encoder network (1) and / or parameters (4c) that characterize the behavior of the decoder arrangement (4) are optimized to the goal (161) of improving the evaluation (5a) by the cost function (5). R.415044 - 18 - 3. Method (100) according to one of claims 1 to 2, wherein • the point cloud of the training example (3) is examined for clusters or accumulations using a given alternative method (151) and • is measured (152) to what extent these clusters or accumulations correspond to the clusters or accumulations (4a) determined by the decoder arrangement (4).

4. Method (100) according to claim 3, wherein the predetermined alternative method comprises k-means clustering into a predetermined number k of clusters (151 a).

5. Method (100) according to one of claims 3 to 4, wherein • bounding boxes are determined from the clusters or accumulations identified using the alternative method (152a), which describe these clusters or accumulations; and • the decoder arrangement (4) is designed (152b) to detect objects that correspond to clusters or accumulations of points.

6. Method (100) according to any one of claims 1 to 5, wherein the encoder network (1) comprises one or more PointNet architectures (121) in which all points of an input point cloud are processed by at least one common multilayer perceptron.

7. Method (100) according to any one of claims 1 to 6, wherein training examples (3) are selected with point clouds (111) whose points represent locations from which a radar or lidar interrogation beam has been reflected.

8. Method (100) according to any one of claims 1 to 7, wherein training examples (3) are selected with point clouds (112) whose measurement data were recorded by at least one sensor carried on a vehicle (50) and / or robot (60).

9. Method (100) according to any one of claims 1 to 8, wherein R.415044 - 19 - • further training examples (6) of point clouds of measurement data are provided (170); • these training examples (6) are converted into vector representations (1 a) using the trained encoder network (1 *) (180); • these vector representations (1 a) are quantized with the trained tokenizer network (2*) (190); • thinned versions (1 a 1 ) of the vectorial representations (1 a) are generated (200) by masking one or more components; • these thinned-out versions (1 a 1) are fed into a transformer network (7) to be trained (210); • with another transformer cost function (8) is measured (220) to what extent the output (7a) of the transformer network to be trained (7) is the same as in the thinned-out versions (1 a 1 ) contains masked components, using the quantized vector representations (2a) for this measurement; and • Parameters (7b) that characterize the behavior of the transformer network (7) to be trained are optimized (230) to improve the evaluation (8a) through the further transformer cost function (8).

10. Method (100) according to claim 9, wherein • further training examples (9) of point clouds of measurement data are provided (240), wherein these further training examples (9) are labelled with respective target outputs (10) that the transformer network (7*) to be trained should generate from them; • these further training examples (9) are processed by the Transformer network to be trained (7*) into outputs (7a) (250); • Deviations of these expenditures (7a) from the respective target expenditures (10) are evaluated using a further predefined task cost function (11) (260) and • Parameters (7b) that characterize the behavior of the transformer network (7*) to be trained are optimized (270) to improve the evaluation (11 a) by the further task cost function (11). R.415044 - 20 - 11. Method (100) according to one of claims 9 to 10, wherein a transformer network (7, 7*) to be trained is selected (211) which is configured to • to assign classification scores to an entered point cloud or parts thereof with respect to one or more classes of a predefined classification; and / or • to semantically segment the input point cloud; and / or • To recognize objects in the input point cloud; and / or • to identify at least one action based on the entered point cloud that performs at least one entity.

12. Method (100) according to any one of claims 9 to 11, wherein • Point clouds (3*) of measurement data (280) recorded with at least one sensor (12) are fed to the trained transformer network (7**); • a control signal (290a) is determined from the output (7a) of the trained transformer network (7**) (290); and • a vehicle (50), a driver assistance system (51), a robot (60), a quality control system (70), a monitoring system (80), and / or a medical imaging system (90), which is controlled by the control signal (290a) (300).

13. Method (100) according to claim 12, wherein the point clouds (3*) cover a spatial environment in which the vehicle (50) or the robot (60) moves, with an extent of at least 50 meters, preferably at least 100 meters, in at least one spatial direction (281).

14. Computer program containing machine-readable instructions which, when executed on one or more computers and / or compute instances, cause the computer(s) and / or compute instance(s) to execute the method (100) according to any one of claims 1 to 13. R.415044 - 21 - 15. Machine-readable data carrier and / or download product containing the computer program according to claim 14.

16. One or more computers and / or compute instances with the computer program according to claim 14, and / or with the machine-readable data carrier and / or download product according to claim 15.

Images