Method and device for training an object detector with an ensemble of several detection heads

Abstract

The invention relates to a computer-implemented method for training a neural network (60), wherein the neural network (60) is designed to detect objects in images (x), and the training comprises the following steps: • determining a feature representation (f) of a training image (x i ) by means of a backbone (B) of the neural network (60); • determining head outputs (C1, C2, C3, P1, P2, P3, S1, S2, S3) based on the feature representation (f) and by means of a plurality (M C , M P , M S ) of heads (H C1 , H C2 , H C3 , H P1 , H P2 , H P3 , H S1 , H S2 , H S3 ) of the neural network (60), wherein the head outputs (C1, C2, C3, P1, P2, P3, S1, S2, S3), in each case for regions of the training image (x i ), characterise a presence or absence of an object or characterise an expansion of the object or characterise a position of the object or characterise the expansion and position; • determining a plurality of first loss values, wherein in each case a first loss value corresponds to a head output (C 1 , C 2 , C 3 , P 1 , P 2 , P 3 , S 1 , S 2 , S 3 ) and characterises a deviation of the head output (C 1 , C 2 , C 3 , P 1 , P 2 , P 3 , S 1 , S 2 , S 3 ) from a desired head output; • adjusting at least one parameter (θ) of the backbone (B) and / or at least one parameter of a head (H C1 , H C2 , H C3 , H P1 , H P2 , H P3 , H S1 , H S2 , H S3 ) based on a second loss value, wherein the second loss value characterises a sum of the first loss values.

Description

[0001]R.401853 - 1 - Description Title Method and device for training an object detector The invention relates to a method for training a neural network, a method for executing the neural network, a device for training, a device for executing, a computer program, and a computer-readable storage medium. State of the art Tran et al. "Hydra: Preserving Ensemble Diversity for Model Distillation", 2020, https: / / arxiv.org / pdf / 2001.04694.pdf discloses an ensemble for image classification. Zhuo et al. "Objects as Points", 2019, https: / / arxiv.org / abs / 1904.07850 discloses CenterNet, a neural network architecture for object detection. Liu et al., "SSD: Single Shot MultiBox Detector," 2016, https: / / arxiv.org / abs / 1512.02325, discloses SSD, a neural network architecture for object detection. Ren et al., "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks," 2015,https: / / arxiv.org / abs / 1506.01497 discloses Faster-RCNN, a neural network architecture for object detection. Advantages of the invention Machine learning systems, especially neural networks, are increasingly used in various fields of technology. These statistical methods have proven particularly suitable in cases where statements can no longer be extracted from data using rules, or not with sufficient quality. R.401853 - 2 - Especially for the detection of objects from digital images, methods from the field of machine learning allow for the currently best possible quality with regard to object detection. In this context, it has been shown thatthat ensemble methods, compared to individual neural networks, improve both the quality of the prediction and the quantification of uncertainty. Ensemble methods can, for example, consist of several independently trained neural networks (deep ensemble). By calculating a plurality of, in particular, different predictions for the same input data, the variance of the predictions can be used, among other things, as an indicator of uncertainty regarding object detection. For example, if the various neural networks of a deep ensemble largely agree in their predictions, the uncertainty is rather low; if they do not,the uncertainty increases. The prediction quality is also improved by calculating the average across the individual predictions. However, this increase in prediction performance comes at the price of a significantly increased computing requirement. Typically, the need for computing capacity and memory grows linearly with the number of neural networks, which can lead to today's neural networks, even with a small ensemble, that the ensemble can no longer be evaluated or can no longer be evaluated sufficiently quickly, or that the memory requirement is too large for the executing computer. This represents a significant risk, particularly in safety-critical applications, since the objects in the images can no longer be identified quickly enough. Advantageously, a neural network trained according to a method with features of claim 1 enables the advantages of object detection using an ensemble (more accurate predictions,more precise uncertainty determination), while the computing time is only slightly increased. This allows the use of the neural network even in computer architectures that have to deal with very limited computing and storage capacities, such as embedded systems. Disclosure of the Invention In a first aspect, the invention relates to a computer-implemented method for training a neural network, wherein the neural network is configured to detect objects in images, and the training comprises the following steps: ^ Determining a feature representation of a training image by means of a backbone of the neural network; ^ Determining head outputs based on the feature representation and by means of a plurality of heads of the neural network,wherein the head outputs each characterize a presence or absence of an object for regions of the training image, or characterize an extent of the object, or characterize a position of the object, or characterize the extent and the position; ^ Determining a plurality of first loss values, wherein a first loss value corresponds to a head output and characterizes a deviation of the head output from a desired head output; ^ Adjusting at least one parameter of the spine and / or at least one parameter of a head based on a second loss value, wherein the second loss value characterizes a sum of the first loss values. The fact that the neural network is designed to detect objects in images can be understood in particular as meaning that the neural network is designed to process an input image, wherein the input image is the image, in particular a digital image,and the neural network is further configured to determine an output signal that characterizes the presence or absence of objects within the image. In particular, the term "object detection" can thus be understood as the detection of the absence of an object. R.401853 - 4 - The expression "object located in the image" or "object of an image" can be understood in particular to mean that the image depicts the object, i.e., there are a plurality of pixel values ​​within the image that together represent the object. Typically, such objects are the mapping of an object from the real world onto a digital image by means of a corresponding sensor. Optical sensors, in particular a camera, can be used as the sensor. However, images within the meaning of the invention can also be recorded by other sensors, for example, a LIDAR, a radar,an ultrasonic sensor or a thermal camera. Preferably, the output signal can also characterize a spatial extent of at least one object within the image and / or a positioning of the object within the image. As an alternative to the spatial extent within the image and / or positioning within the image, the output signal can also characterize a spatial extent with respect to real dimensions of the object in the real world and / or positioning within a real-world coordinate system. For example, using intrinsic and extrinsic parameters of a camera, the pixels and thus objects of an image can be assigned a coordinate in a predefined real-world coordinate system. The neural network can then be trained in this way instead ofThe object's position and / or extent relative to the width and height of the image in pixels is used to determine the position and / or extent of the object relative to the predefined coordinate system. The neural network itself can be understood as consisting of two parts. The first part of the network comprises the backbone. The term backbone is common in the art. Occasionally, the terms base or base network are used synonymously with the term backbone. The backbone, sometimes also referred to as the feature extractor, can in turn be understood as a neural network. The task of the backbone is to transform the image into the feature representation, on the basis of which object detection can then be carried out. The feature representation is preferably at least one three-dimensional tensor, each with a dimension for width,Height and depth. In this case, feature vectors characterizing regions of the image are located along the depth dimension. R.401853 - 5 - where the region of the image represented by the feature vector is indicated by the width and height dimensions of the tensor. The architecture of the backbone can follow common architectures for neural networks, particularly neural networks for images, for example, residual networks (ResNets) or optical transformers (Vision Transformers or Visual Transformers). The feature representation is processed by a second part of the neural network. This second part includes so-called heads. The heads are each designed to determine a head output for regions of the training image.which characterize the presence or absence of an object, or are each designed to characterize an extent of the object, or are each designed to characterize a position of the object, or are each designed to characterize the extent and the position. In addition to the presence or absence of an object, a class membership of an object determined by a head can also be provided in a head output. This is different from known neural networks, since in known neural networks only one head is provided to predict the presence or absence of an object, or an extent of the object, or a position of the object, or the extent and the position. For example, in the known CenterNet architecture, there is one head for the presence / absence determination,One head for position determination and one head for determining the width and height of the object. In the known Single Shot Detection (SSD) architecture, two heads exist, one of which performs the presence / absence determination and the other head performs the position and extent determination. The structure of a Faster-RCNN is similar to the SSD. In contrast to these known architectures, the neural network presented in this invention determines a plurality of heads for determining the absence / absence or for determining the position or for determining the extent or for determining the extent and position. The heads can therefore be understood as an ensemble designed to determine the presence / absence or the position or the extent or the position R.401853 - 6 - and the extent. The inventors were able to determinethat the introduction of an ensemble leads to a significant increase in the accuracy of object detection. Preferably, a head is again a neural network. The neural network can thus preferably be composed of several subnetworks, namely the backbone and the heads. However, other models from the field of machine learning are also possible as the backbone and heads, for example, support vector machines. The heads of the plurality of heads each determine a head output. For training, these head outputs are each compared with a desired head output to determine an initial loss value. In this way, an initial loss value can be determined for each head based on the head output of the head. This creates a one-to-one correspondence between each head and an initial loss value. The initial loss values ​​determined in this way can then be summed, for example, as a weighted sum.in particular, an equally weighted sum can be combined to form the second loss value. Based on the second loss value, the neural network can then be adapted during training, for example, by means of a gradient descent of the parameters of the neural network with respect to the second loss value. A head output is preferably given in the form of a matrix or a tensor. To determine the first loss values, the same desired head output can be compared with the respective determined head outputs. In the event that the head outputs characterize a presence or absence of objects, in particular a cross-entropy between a head output and the desired head output can be provided as the first loss value. For the other cases, in particular a ^, ^-loss function can be used to determine a first loss value. R.401853 - 7 - In preferred further embodiments, it can also be provided that the second loss value characterizes a negative correlation with regard to the various header outputs. The header outputs can in particular be in the form of matrices or tensors, wherein identical indices of different header outputs each characterize an identical part of the training image. For identical indices, a negative correlation of the values ​​at the corresponding indices can then be determined. The training objective can then advantageously be that the respective values ​​of the header outputs at identical indices have the highest possible negative correlation. However, another metric can also be used as the training objective, which expresses a diversity between the header outputs and is thus to be maximized.This can be understood as a regularization of the head outputs. By regularizing the head outputs, the neural network learns to determine different head outputs with respect to a feature representation, whereby the sum of the initial loss values ​​allows the neural network to continue learning to detect the objects correctly. This can be understood as a balancing of two learning objectives: first, the objective of determining the most diverse head outputs possible, and second, the objective of correctly detecting objects in the training image. The inventors were able to determine that the use of negative correlation as a regularization advantageously further increases the accuracy of the neural network.Technically speaking, all negative correlations of values ​​with the same indices between the head outputs can be summed, or an average value can be determined from the negative correlations, wherein the sum or the average value is included in the second loss value as the negative correlation described above. The negative correlation can in particular be added as a term to the sum of the first loss values ​​in order to determine the second loss value. Preferably, the negative correlation can be multiplied by a predefined value, which can be understood as a hyperparameter of the training. R.401853 - 8 - In various preferred embodiments, a plurality of heads can also be provided for determining the presence / absence, for determining the position, and for determining the extent.This can be understood as meaning that an ensemble exists in the neural network for determining presence / absence, position, and extent—that is, a plurality of pluralities of heads. The inventors were able to determine that by using a plurality of pluralities of heads, the accuracy of the neural network can be further increased.In a further aspect, the invention relates to executing the trained neural network, i.e., a method for determining an output signal using the neural network, the method comprising the following steps: ^ Determining a feature representation of the image using the backbone of the neural network; ^ Determining a plurality of head outputs based on the feature representation and using the plurality of heads of the neural network; ^ Determining a summary of the plurality of head outputs to form one head output; ^ Determining the output signal based on the one head output. The steps for executing the neural network, i.e., the steps during the inference phase, are largely the same as the steps necessary to train the neural network. A key difference, however, lies in determining the output signal.During training, the various head outputs were each output so that each head output can be compared with a desired head output. At inference time, the head outputs determined from the majority of heads are combined into a single head output, and based on this combined head output, the output signal is determined that characterizes object detection of objects in the image. Various methods can be selected for combining. In various embodiments, it is possible for the head outputs to be identical in their form (e.g., matrix or tensor) (e.g., the same number of dimensions and elements in the respective dimensions). This can be particularly the case if the individual heads are identical in their architecture (e.g., the same number of layers with the same number of neurons).the last layers of the heads are the same. In these preferred cases, the average values ​​of elements at the same index positions of the head outputs can be used for summarizing. Alternatively, it is also possible to choose a weighted sum instead of averaging, whereby the respective weights can be understood as hyperparameters of the neural network. In particular, each head output can be assigned a weight which is used for all values ​​of the corresponding head output when forming the weighted sum. The inventors were able to determine that the special design of the network with a backbone and the ensemble of heads advantageously increases the accuracy of the network. Furthermore, the requirements for computing capacity and memory are significantly lower compared to the use of a deep ensemble.In various advantageous embodiments, it can further be provided that the output signal additionally comprises a value that characterizes an uncertainty regarding the summary of the plurality of header outputs. If the header outputs of the plurality of header outputs characterize a classification regarding the presence or absence of objects, the uncertainty can, for example, characterize an uncertainty about the actual presence or absence of an object. If the header outputs of the plurality of header outputs characterize a prediction regarding the extent of an object, the uncertainty can, for example, characterize an uncertainty about the actual extent of the object.If the header outputs of the plurality of header outputs characterize a prediction regarding the position of an object, the uncertainty can, for example, characterize an uncertainty about the actual position of the object. Advantageously, the uncertainty can be used to indicate to a user of the neural network how reliably the neural network estimates its own output. This advantageously gives the user the opportunity to intervene in the case of outputs associated with a high degree of uncertainty in order to reduce or suppress possible incorrect behavior. Since the header outputs are typically in the form of a matrix or a tensor, the uncertainty can be determined, in particular, with regard to each element or a subset of elements of the matrix or tensor. This plurality of uncertainties can then be provided in the output signal.This makes it possible, in particular, to show the user of the neural network which area of ​​the image the object detection has determined which uncertainty. The uncertainty can be expressed as a value determined using a predefined function. In particular, the mutual information can be used as the function. Alternatively, other functions are also possible; for example, an entropy can be determined for each head output and a maximum entropy can be provided as an uncertainty value with regard to the index position for each index position within the head outputs. In further aspects of the invention, a method for determining a control signal of an actuator or a display device based on an image is provided, wherein the control signal is determined based on the output signal.Advantageously, the actuator can be controlled more effectively in this way, since the output signal enables more precise object detection. The control signal can in particular also be determined based on the determined uncertainty. For example, the control signal can be selected such that a user of the device is prompted to take over control of the actuator if the uncertainty regarding the summary of the plurality of head outputs reaches or exceeds a predefined threshold value R.401853 - 11 -. Advantageously, the control of the actuator can thus be made more secure, since decisions with high uncertainty and thus high potential for errors can be corrected by the user. In a further aspect, the invention relates to a device which is configured to carry out the method for determining the output signal or the control signal.The device can be understood in particular as a computer or part of a computer. Embodiments of the invention are explained in more detail below with reference to the accompanying drawings. The drawings show: Figure 1 schematically a neural network at training time; Figure 2 schematically a training system for training the neural network; Figure 3 the neural network at inference time; Figure 4 schematically a structure of a control system for controlling an actuator; Figure 5 schematically an embodiment for controlling an at least partially autonomous robot; Figure 6 schematically an embodiment for controlling a manufacturing system; Figure 7 schematically an embodiment for controlling an access system; Figure 8 schematically an embodiment for controlling a monitoring system; R.401853 - 12 - Figure 9 schematically shows an embodiment for controlling a personal assistant; Figure 10 schematically shows an embodiment for controlling a medical imaging system. Description of the Embodiments Figure 1 shows a neural network (60) configured for training. The neural network (60) comprises a backbone (^), which in turn characterizes a neural network, for example, a vision transformer. The backbone is configured to consist of a training image (^). ^ ), which the neural network receives as input, to determine a feature representation (^). This feature representation is then passed to a plurality (^ ^ ) of heads (^ ^^ , ^ ^^ , ^ ^^ ) of the neural network (60) as input of the heads (^ ^^ , ^ ^^ , ^ ^^ ) transmitted. The heads (^ ^^ , ^ ^^ , ^ ^^) are preferably also neural networks, for example convolutional neural networks (CNN). In the example embodiment, the heads are configured to determine the presence or absence of objects. To do this, the heads (^ ^^ , ^ ^^ , ^ ^^ ) header output (^ ^ , ^ ^ , ^ ^ ) in the form of matrices whose elements correspond to areas of the training image (^ ^ ) correspond. The receptive field of an element of the matrix defines the extent of the area within the training image (^ ^). An element can then characterize whether or not an object is located within the corresponding area of ​​the training image. Preferably, an element can indicate whether a center point of a bounding box enclosing the object lies within the area. The element can preferably be a real-valued number between 0 and 1, which can be characterized as the probability of the presence of an object. From a predefined threshold, e.g., 0.5, an element can then be understood as indicating the presence of an object. Values ​​less than 0.5 in this case characterize the absence of objects within the corresponding area. The threshold can be understood as a hyperparameter of the neural network. R.401853 - 13 - The header outputs (^ ^ , ^ ^ , ^ ^ ) are used in training as part of an output signal (^ ^) is provided. Preferably, the neural network (60) comprises a further plurality (^ ^ ) of heads (^ ^^ , ^ ^^ , ^ ^^ ). The heads (^ ^^ , ^ ^^ , ^ ^^ ) of this further majority (^ ^ ) determine further header output (^ ^ , ^ ^ , ^ ^ ) which characterize the exact position and are reflected in the output signal (^ ^ ) are provided. Alternatively or additionally, the neural network (60) for determining an extent of the object (for example width and height within the image) comprises a further plurality (^ ^ ) of heads (^ ^^ , ^ ^^ , ^ ^^ ). The heads (^ ^^ , ^ ^^ , ^ ^^ ) of this further majority (^ ^ ) determine further header output (^ ^ , ^ ^ , ^ ^) which characterize the exact position and are reflected in the output signal (^ ^ ) are provided. In further embodiments (not shown), it can also be provided that the extent and position are determined jointly by heads of a plurality of heads. Figure 2 shows an embodiment of a training system (140) for training the neural network (60) using a training data set (T). The training data set (T) comprises a plurality of training images (^ ^ ) used to train the neural network (60), wherein the training data set (T) is further assigned to a respective input image (^ ^ ) a desired output signal (^ ^ ) which is connected to the input image (^ ^ ) and the presence or absence of objects within the training image (^ ^ ) as well as their position and extent within the training image (^ ^). For training, a training data unit (150) accesses a computer-implemented database (St2), wherein the database (St2) provides the training data set (T). The training data unit (150) determines, preferably randomly, at least one training image (^ ^ ) and that for the training image (^ ^ ) corresponding desired output signal (^ ^ ) and transmits the training image (^ ^ ) to the neural network (60). The desired output signal (^ ^ ) includes at least one desired head output, for example a desired head output regarding the presence or absence of objects in the training image (^ ^ ). Preferably, the desired output signal (^ ^ ) a desired header output for the presence or absence of objects within the training image (^ ^), for a position of objects within the training image (^ ^ ) and for an extension of the objects within the training image (^ ^ ). The neural network (60) determines based on the input image (^ ^ ) an output signal (^ ^ ). The desired output signal (^ ^ ) and the determined output signal (^ ^ ) are transmitted to a change unit (180). Based on the desired output signal (^ ^ ) and the determined output signal (^ ^ ) new parameters (Φ′) for the neural network (60) are then determined by the change unit (180). For this purpose, the change unit (180) first determines a plurality of first loss values, wherein a loss value represents a deviation of a head output (^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ ) of the output signal (^ ^) from a corresponding one from the desired output signal (^ ^ ) included desired header output. A header output (^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ ) corresponds to a desired header output if and only if the header output (^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ ) describes the same property (either presence or absence, position, extent) as the desired head output. For example, if a head output characterizes the presence or absence of an object, this head output is compared with the desired head output for presence or absence. In particular, the values ​​of the output signal (^ ^ ) included capitation expenses (^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^, ^ ^ , ^ ^ , ^ ^ , ^ ^ ) are each in the form of a matrix or a three-dimensional tensor, in which case the desired output signal (^ ^ ) is present in the form of a matrix or a three-dimensional tensor. The corresponding matrices or tensors contain values ​​R.401853 - 15 - or vectors along a height and a width dimension, respectively, that characterize a presence or absence, a position, or an extent. Values ​​or vectors at equal indices along the height and width dimensions of a header output (^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^) and a corresponding header output can then be compared using a loss function. The values ​​thus determined can be accumulated using a mathematical operation, for example, summed, weighted summed, or their maximum value determined. The result of the accumulation can then be used as a first loss value with respect to the header output (^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ ) are provided. The determined first loss values ​​can then be summarized to determine a second loss value. The summary can again be done by a sum, a weighted sum, or by determining the maximum of the first values. Preferably, it is also possible for the second loss value to be enriched by at least one regularization, for example a ^ ^ - or ^ ^-Normalization of parameters (Φ) of the neural network (60). Preferably, a negative correlation between the head output (^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ ) as a regularization in the second loss value. To do this, an average value or average vector can first be calculated over the elements of the header outputs (^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ ) along the height and width dimensions. A negative correlation can then be determined for each element, and the values ​​thus determined can be summarized into a single negative correlation. These steps can be achieved, in particular, by the formula be characterized, where ^ (^) , … , ^ (^)the plurality of header outputs, ^ the number of header outputs, ^ the number of elements along the height dimension within a header output, ^ the number of elements along the width dimension within a header output, the element of the ^-th R.401853 - 16 - header output at position ℎ, ^ along the height or width dimension and || ⋅ | | ^ is the Euclidean norm. The second loss value can therefore be based on the formula: where ^2 is the second loss value, ^1 (^)is the ^-th first loss value and ^ is a weighting coefficient of the regularization, where ^ characterizes a hyperparameter of the training. The change unit (180) determines the new parameters (Φ′) on the basis of the second loss value. In the exemplary embodiment, this is done using a gradient descent method, preferably Stochastic Gradient Descent, Adam, or AdamW. In further exemplary embodiments, the training can also be based on an evolutionary algorithm or a second-order optimization. The determined new parameters (Φ′) are stored in a model parameter memory (St1). Preferably, the determined new parameters (Φ′) are provided to the neural network (60) as parameters (Φ).In further preferred embodiments, the described training is repeated iteratively for a predefined number of iteration steps or iteratively repeated until the first loss value falls below a predefined threshold. Alternatively or additionally, it is also conceivable for the training to be terminated when an average first loss value with respect to a test or validation data set falls below a predefined threshold. In at least one of the iterations, the new parameters (Φ′) determined in a previous iteration are used as parameters (Φ) of the neural network (60). Furthermore, the training system (140) can comprise at least one processor (145) and at least one machine-readable storage medium (146), R.401853 - 17 - which contains instructions which, when executed by the processor (145), cause the training system (140) to carry out a training method according to one of the aspects of the invention. Figure 3 shows how the neural network (60) can be used for object detection of objects of an input image (^) at inference time. Preferably, parameters (Φ) of the neural network (60) are available in trained form. In other words, the neural network (60) has already been trained at inference time. Since at inference time, not multiple head outputs regarding presence or absence or position or extent are desired, but only one at a time, the neural network (60) at inference time comprises a summarization unit (^. ^ , ^ ^ , ^ ^), which summarizes the header outputs of the plurality of header outputs into a single header output (^, ^, ^). In the exemplary embodiment, it is provided that presence or absence, position and extent are determined based on a respective plurality of header outputs. In this case, one summary unit (^, ^, ^) is provided for each plurality of header outputs in the neural network (60). The correspondingly determined individual header outputs (^, ^, ^) are then made available in an output signal (^) by the neural network (60). In other exemplary embodiments (not shown), it can also be provided that bounding boxes are determined based on the individual header outputs (^, ^, ^), and that the bounding boxes are provided in the output signal (^) instead of the header outputs (^, ^, ^) or in addition to the header outputs (^, ^, ^).Figure 4 shows a control system (40) which is designed to determine a control signal (A) of an actuator (10) based on the neural network (60). At preferably regular time intervals, the environment (20) of the actuator (10) is detected by a sensor (30), in particular a camera sensor, which can also be provided by a plurality of camera sensors, for example a stereo camera. The sensor signal (S) - or in the case of multiple sensors, one sensor signal (S) each - of the sensor (30) is transmitted to the control system (40). The control system (40) thus receives a sequence of sensor signals (S). The control system (40) determines control signals (A) from these, which are transmitted to the actuator (10). R.401853 - 18 - The control system (40) receives the sequence of sensor signals (S) from the sensor (30) in an optional receiving unit (50), which converts the sequence of sensor signals (S) into a sequence of input images (^) (alternatively, the sensor signal (S) can also be adopted directly as the input image (^)). The input image (^) can, for example, be a section or further processing of the sensor signal (S). In other words, the input image (^) is determined depending on the sensor signal (S). The sequence of input images (^) is fed to the neural network (60). The neural network (60) is preferably parameterized by parameters (Φ) that are stored in a parameter memory (DB) and provided by the latter. The neural network (60) determines output signals (^) from the input images (^).The output signals (^) are fed to an optional conversion unit (80), which determines control signals (A) therefrom, which are fed to the actuator (10) in order to control the actuator (10) accordingly. The actuator (10) receives the control signals (A), is controlled accordingly, and performs a corresponding action. The actuator (10) can comprise a (not necessarily structurally integrated) control logic, which determines a second control signal from the control signal (A), which is then used to control the actuator (10). In further embodiments, the control system (40) comprises the sensor (30). In yet further embodiments, the control system (40) alternatively or additionally also comprises the actuator (10).In further preferred embodiments, the control system (40) comprises at least one processor (45) and at least one machine-readable storage medium (46) on which instructions are stored which, when executed on the at least one processor (45), cause the control system (40) to carry out the method according to the invention. R.401853 - 19 - In alternative embodiments, a display device (10a) is provided as an alternative or in addition to the actuator (10). The display device (10a) can in particular be an optical display device, for example a screen. Alternatively, the display device (10a) can also be designed for other human senses. It can, for example, be a loudspeaker or indicate an event to a user via haptic signals such as vibration.Figure 5 shows how the control system (40) can be used to control an at least partially autonomous robot, here an at least partially autonomous motor vehicle (100). The sensor (30) can, for example, be a camera sensor, preferably arranged in the motor vehicle (100). In the exemplary embodiment, the neural network (60) is configured to identify objects recognizable in the input images (^), for example other vehicles, people, animals, or static objects such as traffic lights or houses. The actuator (10), which is preferably arranged in the motor vehicle (100), can, for example, be a brake, a drive, or a steering system of the motor vehicle (100).The control signal (A) can then be determined in such a way that the actuator or actuators (10) are controlled in such a way that the motor vehicle (100), for example, prevents a collision with the objects identified by the neural network (60), in particular if the objects are of a certain class, e.g. pedestrians. Alternatively or additionally, the display device (10a) can be controlled with the control signal (A). The display device can, for example, be a screen on which the identified objects are shown. It is also possible for the display device (10a) to be controlled with the control signal (A) in such a way that it emits a visual or acoustic warning signal if it is determined that the motor vehicle (100) is at risk of colliding with one of the identified objects.The warning by means of a warning signal can also be provided by means of a haptic warning signal, for example by vibrating a steering wheel of the motor vehicle (100). R.401853 - 20 - Alternatively, the at least partially autonomous robot can also be another mobile robot (not shown), for example one that moves by flying, swimming, diving, or walking. The mobile robot can also be, for example, an at least partially autonomous lawnmower or an at least partially autonomous cleaning robot. In these cases, too, the control signal (A) can be determined such that the drive and / or steering of the mobile robot are controlled such that the at least partially autonomous robot prevents, for example, a collision with objects identified by the image neural network (60).Figure 6 shows an embodiment in which the control system (40) is used to control a production machine (11) of a production system (200) by controlling an actuator (10) controlling the production machine (11). The production machine (11) can be, for example, a machine for punching, sawing, drilling and / or cutting. It is further conceivable that the production machine (11) is designed to grip a manufactured product (12a, 12b) by means of a gripper. The sensor (30) can then be, for example, a video sensor which detects, for example, the conveying surface of a conveyor belt (13), wherein manufactured products (12a, 12b) can be located on the conveyor belt (13). The neural network (60) can, for example, be configured to detect the manufactured products (12a, 12b) on the conveyor belt and, in particular, to determine their position on the conveyor belt.The actuator (10) controlling the production machine (11) can then be controlled depending on the determined positions of the manufactured products (12a, 12b). For example, the actuator (10) can be controlled such that it punches, saws, drills, and / or cuts a manufactured product (12a, 12b) at a predetermined location on the manufactured product (12a, 12b). Furthermore, it is conceivable that the neural network (60) is configured to determine further properties of a manufactured product (12a, 12b) alternatively or in addition to the position. In particular, it is conceivable that the neural network R.401853 - 21 - (60) determines whether a manufactured product (12a, 12b) is defective and / or damaged. In this case, the actuator (10) can be controlled in such a way that the production machine (11) sorts out a defective and / or damaged manufactured product (12a, 12b).Figure 7 shows an embodiment in which the control system (40) is used to control an access system (300). The access system (300) can comprise a physical access control, for example a door (401). The sensor (30) can in particular be a video sensor or thermal imaging sensor configured to detect an area in front of the door (401). A detected image can be interpreted by means of the neural network (60). In particular, the neural network (60) can detect people in a transmitted input image (^). If several people have been detected simultaneously, the identity of the people can be determined particularly reliably by associating the people (i.e., the objects) with one another, for example by analyzing their movements. The actuator (10) can be a lock that, depending on the control signal (A), enables or disables the access control, for example, opens or disables the door (401).For this purpose, the control signal (A) can be selected depending on the output signal (^) determined for the input image (^) by means of the neural network (60). For example, it is conceivable that the output signal (^) comprises information that characterizes the identity of a person detected by the neural network (60), and the control signal (A) is selected based on the identity of the person. Instead of physical access control, logical access control can also be provided. Figure 8 shows an embodiment in which the control system (40) is used to control a surveillance system (400). This embodiment differs from the embodiment shown in Figure 7 in that the display device (10a) is provided instead of the actuator (10), which is controlled by the control system (40). For example, the sensor (30) can record an input image (^) on which at least one person R.401853 - 22 - can be recognized, and the position of the at least one person is detected by means of the neural network (60). The input image (^) can then be displayed on the display device (10a), wherein the detected persons can be highlighted in color. Figure 9 shows an embodiment in which the control system (40) is used to control a personal assistant (250). The sensor (30) is preferably an optical sensor that receives images of a gesture of a user (249), for example a video sensor or a thermal imaging camera. Depending on the signals from the sensor (30), the control system (40) determines a control signal (A) of the personal assistant (250), for example by the neural network (60) performing gesture recognition. This determined control signal (A) is then transmitted to the personal assistant (250), who is thus controlled accordingly.The determined control signal (A) can in particular be selected such that it corresponds to a presumed desired control by the user (249). This presumed desired control can be determined depending on the gesture recognized by the neural network (60). The control system (40) can then, depending on the presumed desired control, select the control signal (A) for transmission to the personal assistant (250) and / or select the control signal (A) for transmission to the personal assistant according to the presumed desired control (250). This corresponding control can, for example, include the personal assistant (250) retrieving information from a database and reproducing it in a manner that is receptive to the user (249).Instead of the personal assistant (250), a household appliance (not shown), in particular a washing machine, a stove, an oven, a microwave or a dishwasher, can also be provided in order to be controlled accordingly. R.401853 - 23 - Figure 10 shows an embodiment in which the control system (40) is used to control a medical imaging system (500), for example an MRI, X-ray or ultrasound device. The sensor (30) can be provided, for example, by an imaging sensor. The control system (40) controls the display device (10a). The display device (10a) can, in particular, be provided by a screen. The sensor (30) is configured to acquire an image of a patient, for example an X-ray image, an MRI image or an ultrasound image. At least part of the image is transmitted to the neural network (60) as an input image (^).The neural network (60) can, for example, be configured to detect different types of tissue that can be identified in the input image (^). The control signal (A) can then be selected such that the determined types of tissue are highlighted in color on the display device (10a). In further embodiments (not shown), the imaging system (500) can also be used for non-medical purposes, for example, to determine material properties of a workpiece. For this purpose, the imaging system (500) can record an image of a workpiece. In this case, the neural network (60) can be configured to receive at least part of the image as input image (^) and detect materials of the workpiece. The detection determined in this way can, for example, be displayed together with the input image on the display device (10a).The term "computer" encompasses any device for executing predefined calculation instructions. These calculation instructions can be in the form of software, or in the form of hardware, or even a mixture of software and hardware. In general, a plurality can be understood as indexed, i.e., each element of the plurality is assigned a unique index, preferably by assigning consecutive integers to the elements contained in the plurality. Preferably, when a plurality comprises ^ elements, R.401853 - 24 - where ^ is the number of elements in the plurality, the elements are assigned the integers from 1 to ^.

Claims

R.401853 - 25 - Claims 1. Computer-implemented method for training a neural network (60), wherein the neural network (60) is designed to detect objects in images (^), and the training comprises the following steps: ^ Determining a feature representation (^) of a training image (^ ^ ) by means of a backbone (^) of the neural (60); ^ Determining head outputs (^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ ) based on the feature representation (^) and by means of a plurality (^ ^ , ^ ^ , ^ ^ ) of heads (^ ^^ , ^ ^^ , ^ ^^ , ^ ^^ , ^ ^^ , ^ ^^ , ^ ^^ , ^ ^^ , ^ ^^ ) of the neural network (60), where the head outputs (^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^^ , ^ ^ ) for each area of ​​the training image (^ ^ ) characterizes a presence or absence of an object or characterizes an extent of the object or characterizes a position of the object or characterizes the extent and the position; ^ Determining a plurality of first loss values, wherein in each case a first loss value is associated with a header output (^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ ) and a deviation of the header output (^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ ) characterized by a desired head output; ^ Adjusting at least one parameter (^) of the backbone (^) and / or at least one parameter of a head (^ ^^ , ^ ^^ , ^ ^^ , ^ ^^ , ^ ^^, ^ ^^ , ^ ^^ , ^ ^^ , ^ ^^ ) based on a second loss value, wherein the second loss value characterizes a sum of the first loss values.

2. The method according to claim 1, wherein the second loss value additionally characterizes a negative correlation between the head outputs (^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ ) characterized. R.401853 - 26 - 3. The method according to claim 1 or 2, wherein the neural network (60) comprises a plurality of pluralities (^ ^ , ^ ^ , ^ ^ ) of heads (^ ^^ , ^ ^^ , ^ ^^ , ^ ^^ , ^ ^^ , ^ ^^ , ^ ^^ , ^ ^^ , ^ ^^ ) and for each plurality (^ ^ , ^ ^ , ^ ^ ) of heads (^ ^^ , ^ ^^ , ^ ^^ , ^ ^^ , ^ ^^ , ^ ^^ , ^^^ , ^ ^^ , ^ ^^ ) a second loss value is determined in each case and the neural network (60) is trained based on a sum of the second loss values.

4. A method for determining an output signal (^) by means of a neural network (60), wherein the neural network is set up according to one of claims 1 to 3 and the output signal (^) characterizes the presence or absence of objects within an image (^), the method comprising the following steps: ^ determining a feature representation (^) of the image (^) by means of the backbone (B) of the neural network (60); ^ determining a plurality of head outputs based on the feature representation (^) and by means of the plurality (^ ^ , ^ ^ , ^ ^ ) of heads (^ ^^ , ^ ^^ , ^ ^^ , ^ ^^ , ^ ^^ , ^ ^^ , ^ ^^ , ^ ^^ , ^ ^^) of the neural network (60); ^ Determining a summary (^ ^ , ^ ^ , ^ ^ ) the majority of headed expenditures (^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^ , ^ ^) to a head output (^, ^, ^); ^ Determining the output signal (^) based on the one head output (^, ^, ^).

5. The method according to claim 4, wherein the method additionally comprises training the neural network according to one of claims 1 to 3.

6. The method according to claim 4 or 5, wherein the output signal (^) additionally comprises a value that characterizes an uncertainty regarding the combination of the plurality of head outputs.

7. A method for determining a control signal (A) of an actuator (10) or a display device (10a) based on an image (^), wherein in the method an output signal (^) is determined based on an image (^) and according to one of claims 4 to 6 and subsequently the control signal R.401853 - 27 - (A) is determined based on the output signal (^).

8. Device (40) configured to carry out the method according to one of claims 4 to 7.

9. Training device (140) configured to carry out the method according to one of claims 1 to 3.

10. Computer program configured to carry out the method according to one of claims 1 to 7 when executed by a processor (45, 145).

11. Machine-readable storage medium (46, 146) on which the computer program according to claim 10 is stored.

Images