Device and method for determining semantic segmentation and / or instance segmentation of image

Abstract

To provide a computer-implemented method for determining an output signal (y) characterizing a semantic segmentation and / or an instance segmentation of an image (x).SOLUTION: A method comprises the steps of: determining a first intermediate output signal from a machine learning system (60), wherein the first intermediate output signal characterizes a semantic segmentation and / or an instance segmentation of the image (x); adapting parameters (Φ) of the machine learning system (60) based on a loss function, wherein the loss function characterizes an entropy or a cross-entropy of the first intermediate output signal (o1); and determining the output signal (y) from the machine learning system (60) based on the image (x) and the adapted parameters (Φ).SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a method for determining an output signal by a machine learning system, a machine learning system, a control system for controlling an actuator, a computer program, and a machine-readable storage medium.

Background Art

[0002] Prior Art “Test-Time Adaptation to Distribution Shift by Confidence Maximization and Input Transformation” by Mummadi et al., June 28, 2021 (available online: https: / / arxiv.org / pdf / 2106.14999.pdf) discloses a method for test-time adaptation.

[0003] Technical Background Machine learning (ML) systems, particularly neural networks, have increasingly been used in a growing number of technical fields, especially for analyzing sensor data from optical sensors.

[0004] When an ML system is deployed, the data encountered after deployment may differ from the typical data used for training. In this case, a domain shift (also known as a distribution shift) occurs. The performance of the ML system, such as classification accuracy or the accuracy of continuous values ​​determined in regression, can deteriorate significantly after a domain shift. Therefore, it is desirable to devise methods for adapting the ML system to the new domain and / or distribution. Such adaptation methods are particularly desirable in situations where reliable annotations for the new domain and / or distribution cannot be used during inference. Specific examples include "source-free domain adaptation" or "test-time adaptation," in which adaptation is performed based solely on unlabeled data from the target domain (the shifted domain), without requiring the data on which the network was initially trained (the source domain). [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] “Test-Time Adaptation to Distribution Shift by Confidence Maximization and Input Transformation,” by Mummadi et al., June 28, 2021 (Available online: https: / / arxiv.org / pdf / 2106.14999.pdf) [Overview of the Initiative] [Means for solving the problem]

[0006] Disclosure of the invention According to a first aspect, the present invention relates to a computer-implemented method for determining an output signal (y) representing the semantic segmentation and / or instance segmentation of an image, wherein the method The steps include: determining a first intermediate output signal using a machine learning system, wherein the first intermediate output signal represents the semantic segmentation and / or instance segmentation of the image; The steps include: adapting the parameters of a machine learning system based on a loss function, wherein the loss function represents the entropy or cross-entropy of a first intermediate output signal; The steps include determining the output signal from the machine learning system based on the image and adapted parameters, Includes.

[0007] Images can be acquired particularly by sensors, such as cameras, LiDAR sensors, radar sensors, ultrasonic sensors, or thermal cameras. However, images can also be generated by computers, for example, by methods for rendering virtual scenes on a computer, or by machine learning models, such as generative adversarial networks.

[0008] The image is processed via a machine learning system. In other words, the machine learning system takes the image as input. The machine learning system may preferably be a neural network configured to determine an intermediate signal. Selectively, the machine learning system may include a machine learning model, for example, a neural network for determining a first intermediate output, i.e., the machine learning model can be configured to perform a method for determining an output signal.

[0009] Whether contained within a machine learning model or being a machine learning model itself, a neural network can be, in particular, a convolutional neural network, a visual transformer, or a vision transformer.

[0010] This method is particularly suitable for execution during the inference time of a machine learning system. Inference time can also be considered as the "post-deployment" or "post-training" period for the machine learning system. The first intermediate output signal can be modified by this method to favorably adapt the machine learning system to images from its training data, i.e., to the data the machine learning system experiences during inference. The inventors have found that this adaptation leads to an improvement in the predictive performance of the machine learning system. In relation to semantic segmentation or instance segmentation, predictive performance can generally be determined based on a metric known as mean intersection over union (also known as mIoU).

[0011] This method can also be understood as a method for adapting to test time. Adapting to test time is particularly suitable when the domain from which the training dataset is generated is different from the domain from which the images are generated.

[0012] Semantic segmentation and / or instance segmentation can, in particular, assign classifications to pixels in an image. These classifications can be given in the form of an index of the class to which each pixel belongs. Preferably, the classification is given as a vector of probabilities for multiple classes. This is also known as soft classification. Semantic segmentation or instance segmentation can also be considered as a label map.

[0013] Semantic segmentation and / or instance segmentation can be given in particular in the form of tensors. Each tensor can be a three-dimensional tensor. Preferably, the first and second dimensions of the tensor represent the height and width of the semantic or instance segmentation, respectively, while the third dimension of the tensor represents multiple classes. This can also be understood as the tensor containing multiple classifications, in which case the classifications are indexed along the width and height dimensions. Preferably, the tensor contains the same number of elements along the width and height dimensions as the image contains along its width and height, respectively. Thus, one classification can be understood as corresponding to one pixel; that is, one classification located at height i and width j can represent one pixel of the image at height i and width j. Thus, the expression "classification at position i,j" can be understood as the classification being located at height i and width j.

[0014] The first intermediate output signal can be understood as representing semantic segmentation and / or instance segmentation, and if no adaptation is performed, it can be output by the machine learning system. To adapt the machine learning system to the image, this method can be understood as determining the adapted parameters of the machine learning system, i.e., new parameters. In this case, the machine learning system uses the new parameters adapted to the image to determine the output for the image. In other words, the output signal is determined similarly to the first intermediate output signal, but based on the new parameters of the machine learning system.

[0015] Semantic segmentation and / or instance segmentation can also represent the classification portion of object detection. For example, a common machine learning system for single-shot object detection (also known as single-shot detection or SSD) divides the object detection task into several subtasks. The first subtask may be to classify whether a pixel in an image represents the presence of an object or a specific part of an object, the second subtask may be to determine the exact location of the object, and the third subtask may be to determine the extent of the object, e.g., width and height. This approach is also followed for common object detectors such as CenterNet. Therefore, since the subtask of classifying the presence of an object is essentially a semantic segmentation task, this method can also be applied to the object detection task, at least to solve this subtask. It should be noted here that approaches like CenterNet do not determine the classification for every pixel, but rather for patches of pixels in an image, e.g., 4x4 patches. However, since each pixel can still be understood as corresponding to a single classification, even though the classification is shared with other pixels, the method proposed here can still be used.

[0016] The step of determining new parameters can be achieved, in particular, by a gradient descent algorithm, especially if the machine learning system is or includes a neural network. The inventors have found that the adaptation of parameters in a machine learning system can be advantageously achieved based on a single image, i.e., on this image alone. This is an advantage over known methods that require multiple images, i.e., a batch of multiple images, for adaptation, because otherwise the machine learning system would overfit to that image. The inventors have found that, due to its ability to perform semantic segmentation, the machine learning system is given multiple classifications used in gradient descent, which allows the use of a single image for parameter adaptation without overfitting to that image.

[0017] This method can also be applied to sequences of multiple images, such as videos. In this case, the method can be applied individually to each image, i.e., iteratively. In each iteration, the parameters determined in the previous iteration can be used as a starting point for finding new parameters for the machine learning system based on the images in that iteration. Selectively, it is also possible to use the parameters determined when training the machine learning system on its training dataset, i.e., to reset the parameters at the start of each iteration.

[0018] Preferably, the loss function represents the average entropy of the classification obtained for each pixel of the image.

[0019] In other words, the loss function can represent the entropy determined for each classification, represented by the semantic segmentation and / or instance segmentation represented by the first output signal, and in this case, the entropy can be averaged for the purpose of determining the entropy of the semantic segmentation or instance segmentation.

[0020] The loss function can be, for example, the negative of the average of the individual Shannon entropies determined for each classification represented by the semantic segmentation and / or instance segmentation represented by the first intermediate output signal. Preferably, the negative of the average of the individual Shannon entropies is one of a plurality of terms, in which case the loss function is the sum of those terms, or is represented by the sum of those terms.

[0021] It is also possible for the loss function to represent the cross-entropy between the first intermediate output signal and the second intermediate output signal. Preferably, the second intermediate output signal is determined using a conversion function based on the first intermediate output signal, where the second intermediate output signal represents the semantic segmentation and / or instance segmentation of the image. For example, the first intermediate output signal can be the output of a machine learning model, such as a neural network, where the conversion function processes the determined first intermediate output signal to determine the second intermediate output signal. The conversion function can also be part of a machine learning model, such as a layer of a neural network, in which case the output of this layer is the second intermediate output signal and the input to this layer is the first intermediate output layer.

[0022] In the following, the expression "classification in the first intermediate output signal" can be understood as a classification represented by semantic segmentation and / or instance segmentation represented by the first intermediate output signal. Similarly, the expression "classification in the second intermediate output signal" can be understood as a classification represented by semantic segmentation and / or instance segmentation represented by the second intermediate output signal.

[0023] The cross-entropy between the first intermediate output signal and the second intermediate output signal can be determined in the same manner as the entropy. Each classification in the first intermediate output signal has a corresponding classification in the second intermediate output signal, that is, the second intermediate output signal has a classification having the same position as that in the first intermediate output signal. In this case, the cross-entropy can be determined for each pair, preferably for all pairs, of the classification in the first intermediate output signal and the corresponding classification in the second intermediate output signal. The loss term may also include the hard likelihood ratio or soft likelihood ratio of the corresponding classification instead of the cross-entropy.

[0024] The conversion function can be defined a priori. For example, the conversion function can represent an edge-preserving smoothing filter, particularly a bilateral filter. What the inventors have found is that this type of filtering advantageously removes misclassifications in the semantic segmentation or instance segmentation represented by the first intermediate output signal, thereby further enhancing the performance of the machine learning system.

[0025] Optionally, the conversion function can be a trainable function, for example, it can be one layer of the neural network disclosed above.

[0026] Additionally, the loss function may further represent the likelihood, preferably log-likelihood, of at least a portion of the first intermediate output signal, in which case this likelihood is determined based on a density model of at least a portion of the image.

[0027] The density model can, in particular, represent the maximum likelihood estimate of classifications from semantic segmentation and / or instance segmentation from a dataset of a machine learning system, preferably a training dataset. In this way, the machine learning system can adapt its output while still supplying an output signal that represents semantic segmentation and / or instance segmentation that is reasonable with respect to known semantic segmentation and / or instance segmentation. In other words, the output represents semantic segmentation and / or instance segmentation that better matches the semantic segmentation and / or instance segmentation from the dataset. The inventors have found that this advantageously leads to further improvements in the performance of the machine learning system.

[0028] Density models can model the overall occurrence of a class. For example, a model can be determined by first creating individual class histograms of the semantic segmentation and / or instance segmentation contained in a dataset. In this case, the histograms can be used to fit the density model, for example, by maximum likelihood estimation. Density models can be represented by multinomial models, Gaussian models, or mixture models such as Gaussian mixture models or multinomial mixture models. Histograms can be normalized, for example, by L1 normalization before being used for fitting the density model.

[0029] Instead of representing the classification of the entire first intermediate output signal, the likelihood expressed by the loss function can also represent the average likelihood of multiple patches, where these patches are determined based on the first intermediate output signal.

[0030] A patch can be understood as a rectangular region of the first intermediate output signal along its width and height dimensions. Thus, a patch can represent a subset of the classifications contained in the first intermediate output signal. For semantic segmentation and / or instance segmentation of a dataset, multiple patches of comparable size can be extracted and used to fit density models, which can also be polynomial, Gaussian, or mixed models. During adaptation, multiple patches can be extracted from the first intermediate output signal, and a likelihood can be determined for each extracted patch. Patch extraction can be particularly random; for example, the location from which the patches are extracted can be determined randomly. In this case, a likelihood can be determined from the density model for each patch, and the likelihoods thus determined can be averaged and used as a term in the loss function.

[0031] The inventors found that using patches in this way could further improve the performance of machine learning systems.

[0032] The likelihood for a patch can also be determined by using a feature extractor to determine the feature representation of this patch and supplying the likelihood of the feature representation as the likelihood of this patch, where the likelihood of the feature representation is determined by a density model.

[0033] The feature extractor can be a neural network, such as a convolutional neural network. Preferably, the parameters of the neural network are determined when determining the gradients for adapting the parameters of the machine learning system. Thus, the feature extractor can be understood as having fixed parameters. To determine the density model, patches can also be extracted from the semantic segmentation and / or instance segmentation of the dataset. In this case, the individual feature representations for the extracted patches can be determined based on the feature extractor. In this case, the feature representations can be used to fit the density model, for example, by using maximum likelihood estimation. To adapt the machine learning system, multiple patches can also be randomly selected and passed through the feature extractor. In this case, the resulting feature representations can be fed into the density model, which then determines multiple likelihood values ​​or log-likelihood values. In this case, the likelihood values ​​or log-likelihood values ​​can be averaged to determine the likelihood value represented by the loss function.

[0034] In each individual embodiment, the likelihood expressed by the loss function, using information from semantic segmentation and / or instance segmentation from the dataset, can be used, in particular, in one loss term of the loss function, which is expressed by the formula

number

[0035] When using multiple patches, the classifications included in a patch can be modified to indicate to the machine learning system which combination of classifications within a patch is unexpected and therefore should not be output in the output signal. For example, the classifications can be swapped within each patch, such as swapping the most common classification in a patch with the second most common classification, or swapping the third most common classification with the fourth most common classification. In this case, the loss function may include a term that incentivizes the machine learning system not to output an output signal that looks like these modified patches. Therefore, the loss function is preferably expressed by equation

number

[0036] Preferably, to determine the output signal, the machine learning system may include a normalization transformation, and the loss may further represent the Kullback-Leibler divergence between the output of this normalization transformation and a predefined probability distribution.

[0037] The phrase "to determine the output signal" can be understood insofar as the normalization transformation has a decisive influence in determining the output signal from an image. For example, when a neural network is used to determine the first intermediate output signal and / or the second intermediate output signal, the normalization layer of the neural network can be understood as the normalization transformation. The normalization layer can be, for example, a batch normalization layer, a group normalization layer, or a layer normalization layer.

[0038] Normalization transforms can, in particular, normalize the input to the normalization transform according to a predefined scheme. For example, in a normalization layer, the input or a portion of the input can be normalized such that they have a zero mean and unit variance. For example, in the case of batch normalization, the input can be a tensor representing multiple images in a batch used to train a machine learning system. In this case, the channels of the tensor can be normalized such that each channel in the tensor has a zero mean and unit variance for all samples in the batch. For inference, the mean and variance with respect to the batch used in training can be averaged, for example, using a moving average. The normalization layer may also include other operations such as scaling and shifting. However, for the purposes of this method, the output mentioned in the loss representing the Kullback-Leibler divergence between the output of the normalization transform and a predefined probability distribution can, in particular, be considered the output of the layer after normalization and before scaling and shifting operations.

[0039] The inventors found that including a loss term representing the Kullback-Leibler divergence helps in adaptation because it further improves the performance of the machine learning system.

[0040] Preferably, the predefined probability distribution is represented by a standard normal distribution. The predefined probability distribution can also be a multivariate standard normal distribution, i.e., a multivariate normal distribution with a null vector as the expected value and an identity matrix as the covariance matrix.

[0041] The Kullback-Leibler divergence can, for example, be included in one term of the loss function, where the loss term is expressed as:

number

number

[0042] A machine learning system may have multiple normalization transformations, for example, multiple normalization layers. In this case, the loss term can be adapted to take into account multiple, preferably, such normalization transformations. That is,

number

[0043] Embodiments of the present invention will be described in further detail with reference to the following drawings. [Brief explanation of the drawing]

[0044] [Figure 1] This is a diagram of a machine learning system. [Figure 2] This figure shows a control system that includes a classifier for controlling actuators within their environment. [Figure 3] This figure shows a control system for controlling a robot that is at least partially autonomous. [Figure 4]This is a diagram showing a control system for controlling manufacturing machinery. [Figure 5] This is a diagram showing a control system that controls an imaging system. [Modes for carrying out the invention]

[0045] Description of the Embodiment Figure 1 shows one embodiment of a machine learning system (60). The machine learning system (60) is configured to accept an image (x) as input. In this embodiment, the image (x) is transferred to a neural network (61) of the machine learning system, which is configured to determine semantic segmentation and / or instance segmentation for the image (x). The neural network can be a convolutional neural network, a vision transformer, or a visual transformer, in particular. In further embodiments (not shown), other machine learning models can be used instead of the neural network (61). The neural network includes several parameters (Φ), such as weights, biases, or normalization parameters. Before use, the neural network (61) can be trained to predict semantic segmentation and / or instance segmentation according to known methods.

[0046] The neural network (61) is configured to determine a first intermediate output signal (o1), which represents the semantic segmentation and / or instance segmentation of an image (x). The first intermediate output signal (o1) is forwarded to an adaptive unit (62). The adaptive unit (62) determines parameters adapted for the neural network (61) based on the first intermediate output signal (o1). The neural network (61) then determines a new first intermediate output signal (o1) based on the image (x) and the adapted parameters. In yet another embodiment (not shown), this new first intermediate output signal (o1) can then be supplied as the output signal (y) of a machine learning system (60).

[0047] To determine the adapted parameters, the adaptive unit (62) can preferably adapt the parameters (Φ) of the neural network (61) based on gradient descent. The loss function for performing gradient descent is: loss term

number

[0048] Preferably, the machine learning system (60) includes a transformation function (g) configured to process the first intermediate output signal (o1) in order to determine a second intermediate output signal (o2). The transformation function (g) can, in particular, represent an edge-preserving smoothing filter, such as a bilateral filter.

[0049] Preferably, the conversion function (g) stores the width and height of the first intermediate output signal (o1) in the second intermediate output signal (o2).

[0050] In this case, the first intermediate output signal (o1) and the second intermediate output signal (o2) can be used as inputs to the cross-entropy loss term in the loss function. The cross-entropy loss term is given by:

number

number

[0051] Preferably, the loss further includes a loss term representing the deviation of a second intermediate output signal (o2) from the leading distribution of the classification. In embodiments where the second intermediate output signal (o2) is not calculated, the deviation between the first intermediate output signal (o1) and the leading distribution can also be determined. The leading distribution can be determined based on label maps from a dataset, e.g., a training dataset for a neural network (61). For each label map, a classification histogram can be determined. The histogram can be optionally normalized, for example, by L1 normalization. Subsequently, a model representing the probability density function can be fitted to the histogram, preferably a mixture model, e.g., a Gaussian mixture model, or, if normalization is not performed, a mixture polynomial model. In this case, an additional term of the loss function

number

[0052] The neural network (61) may optionally include a normalization layer (NL) for determining a first intermediate output signal (o1). The normalization layer (NL) may be, for example, a group normalization layer, a batch normalization layer, or a layer normalization layer. If the neural network (61) includes a normalization layer (NL), the loss is preferably a loss term representing the deviation of the output of the normalization layer (NL) from a multivariate standard normal distribution.

number

number

[0053] It is also possible for the neural network (61) to include multiple normalization layers (NL). In this case, the loss term may represent the sum of loss terms determined individually for each normalization layer (NL), preferably a weighted sum.

[0054] Therefore, the loss function is preferably given by the formula

number

[0055] Instead of the usual summation, the loss can also be determined according to a weighted sum of the individual terms. It is also possible that only one or two of these terms are used for adaptation.

[0056] Figure 2 shows one embodiment of a control system (40) that controls an actuator (10) within its environment (20) based on the output of a machine learning system (60). The actuator (10) interacts with the control system (40). The actuator (10) and its environment (20) together are referred to as the actuator system. Preferably, at equally spaced intervals, an optical sensor (30) senses the state of the actuator system. The sensor (30) may include multiple sensors, such as a stereo camera. The output signal (S) of the sensor (30) encoding the sensed state (or, if the sensor (30) includes multiple sensors, the output signal (S) of each sensor) is transmitted to the control system (40).

[0057] As a result, the control system (40) receives a stream of sensor signals (S). The control system (40) then calculates a series of control signals (A) depending on the stream of sensor signals (S), and these control signals (A) are then transmitted to the actuator (10).

[0058] The control system (40) receives a stream of sensor signals (S) from the sensor (30) in an optional receiving unit (50). The receiving unit (50) converts the sensor signals (S) into an input image (x). Selectively, if the receiving unit (50) is not provided, each sensor signal (S) can be treated as an input image (x) as is. The input image (x) can be realized, for example, as an excerpt from the sensor signals (S). Selectively, the sensor signals (S) can be processed to generate the input image (x). In other words, the input image (x) is supplied according to the sensor signals (S).

[0059] The input image (x) is then sent to the machine learning system (60).

[0060] The machine learning system (60) is parameterized by parameters (Φ), which are stored in and supplied by a parameter memory (P).

[0061] The machine learning system (60) determines the output signal (y) from the input signal (x). The output signal (y) is sent to an optional conversion unit (80), which converts the output signal (y) into a control signal (A). The control signal (A) is then sent to the actuator (10) to control the actuator (10) accordingly. Selectively, the output signal (y) can be treated as the control signal (A) as is.

[0062] The actuator (10) receives a control signal (A) and is controlled accordingly to perform an action corresponding to the control signal (A). The actuator (10) may include control logic that converts the control signal (A) into another control signal, and then the actuator (10) is controlled using this other control signal.

[0063] In yet another embodiment, the control system (40) may include a sensor (30). In yet another embodiment, the control system (40) may optionally or additionally include an actuator (10).

[0064] In yet another embodiment, the control system (40) may be configured to control the display (10a) instead of or in addition to the actuator (10).

[0065] Furthermore, the control system (40) may include at least one processor (45) and at least one machine-readable storage medium (46) which stores instructions, which, when executed, cause the control system (40) to carry out a method according to one aspect of the present invention.

[0066] Figure 3 shows an embodiment in which a control system (40) is used to control a robot that is at least partially autonomous, such as a vehicle (100) that is at least partially autonomous.

[0067] The sensor (30) may include one or more video sensors and / or one or more radar sensors and / or one or more ultrasonic sensors and / or one or more LiDAR sensors. Some or all of these sensors are preferably, but not required, incorporated into the vehicle (100).

[0068] Therefore, the input image (x) can display the surroundings of the vehicle (100), and the output signal (y) can represent the semantic segmentation and / or instance segmentation of the surroundings. The semantic segmentation and / or instance segmentation can, in particular, represent stationary elements such as other road users, buildings or signals, and classifications of roads or drivable ground. In this case, the conversion unit (80) can determine a route such that the vehicle moves on ground classified as drivable by the output signal (y), while avoiding collisions with other elements in the vehicle's environment (20). The control signal (A) can then be determined according to this information.

[0069] Preferably, the actuator (10) incorporated into the vehicle (100) can be realized by the vehicle's (100) brakes, propulsion system, engine, drivetrain, or steering.

[0070] A control signal (A) may also be used selectively or additionally to control the display (10a), for example, to display an object detected by a machine learning system (60). Similarly, if the vehicle (100) deviates from an escape route through an environment determined by the conversion unit (80), the control signal (A) may control the display (10a) so that the display (10a) generates a warning signal. The warning signal may be an audible and / or tactile signal, such as vibration of the vehicle's steering wheel.

[0071] In yet another embodiment, the at least partially autonomous robot can be realized by other mobile robots (not shown) that are capable of moving, for example, by flight, swimming, diving, or walking. The mobile robot can be, in particular, an at least partially autonomous lawnmower or an at least partially autonomous cleaning robot. In all of the embodiments described above, control signals (A) can be determined so that the propulsion unit and / or steering and / or brakes of the mobile robot are controlled so that the mobile robot can avoid collisions with identified objects.

[0072] In yet another embodiment, a robot that is at least partially autonomous can be realized by a gardening robot (not shown) which can use sensors (30), preferably optical sensors, to determine the state of plants in the environment (20). An actuator (10) can control a liquid spray nozzle and / or a cutting device, such as a blade. Depending on the identified type and / or state of the plant, the actuator (10) can determine a control signal (A) to spray an appropriate amount of appropriate liquid onto the plant and / or cut the plant.

[0073] In yet another embodiment, a robot that is at least partially autonomous can be realized by an indoor electrical appliance (not shown), such as a washing machine, stove, oven, microwave oven, or dishwasher. A sensor (30), such as an optical sensor, can detect the state of an object being processed by the household appliance. For example, if the indoor electrical appliance is a washing machine, the sensor (30) can detect the state of the laundry inside the washing machine. In this case, the control signal (A) can be determined depending on the detected material of the laundry.

[0074] Figure 4 shows an embodiment in which a control system (40) is used to control, for example, a manufacturing machine (11) of a manufacturing system (200) which is part of a production line, such as a punching machine, cutter, gun drill, or gripper. The manufacturing machine (11) may include a conveying device for moving the manufactured product (12), such as a conveyor belt or assembly line. The control system (40) controls an actuator (10), which in turn controls the manufacturing machine (11).

[0075] The sensor (30) can be implemented, for example, by an optical sensor that detects the characteristics of a manufactured product (12).

[0076] The machine learning system (60) can perform instance segmentation of the manufactured product (12) on the conveyor belt. The conversion unit (80) can then infer the precise position of the manufactured product (12) based on the instance segmentation. The actuator (10) can then be controlled for subsequent manufacturing steps of the manufactured product (12), depending on the determined position of the manufactured product (12). The actuator (10) can be controlled, for example, to cut the manufactured product (12) at a specific location on the product itself.

[0077] Figure 5 shows one embodiment of a medical imaging system (500) controlled by a control system (40). This imaging system may be, for example, an MRI machine, an X-ray machine, or an ultrasound machine. The sensor (30) may be, for example, an imaging sensor that takes at least one image of a patient, for example, at least one image showing various types of body tissue of the patient.

[0078] In this case, the machine learning system (60) can determine the semantic segmentation of at least a portion of the sensed image. Therefore, at least a portion of the image is used as the input image (x) to the machine learning system (60).

[0079] Next, a control signal (A) can be selected according to this classification, thereby controlling the display (10a). For example, the machine learning system (60) can be configured to detect various types of tissue in a sensed image by classifying the tissue displayed in the image as either malignant or benign. This can be done by semantic segmentation of the input image (x) by the machine learning system (60). In this case, for example, the control signal (A) can be determined to display various tissues on the display (10a) by displaying the input image (x) and coloring various regions of the same tissue type with the same color.

[0080] In other embodiments (not shown), the imaging system (500) can be used for non-medical purposes, for example, to determine the material properties of a workpiece. In these embodiments, the machine learning system (60) can be configured to receive an input image (x) of at least a portion of the workpiece, perform semantic segmentation of the input image (x), and thereby classify the material properties of the workpiece. In this case, a control signal (A) can be determined to display not only the input image (x) but also information regarding the detected material properties on the display (10a).

[0081] The term "computer" can be understood to encompass any device for processing predefined arithmetic rules. These arithmetic rules can be in software form, hardware form, or a hybrid of software and hardware.

[0082] Generally, multiple entities can be understood as being subscripted, that is, each element of those multiple entities is assigned a unique subscript, which is preferably done by assigning consecutive integers to the elements contained in those multiple entities. Preferably, if the multiple entities contain N elements, and N is the number of the elements in those multiple entities, then those elements are assigned integers from 1 to N. It can also be understood that elements in multiple entities can be referred to by subscripts.

Claims

1. 1. A computer-implemented method for determining an output signal (y) representing a semantic and / or instance segmentation of an image (x), comprising: - determining a first intermediate output signal from a machine learning system (60), said first intermediate output signal representing a semantic and / or instance segmentation of said image (x); Adapting the parameters (Φ) of the machine learning system (60) based on a loss function, the loss function being a function of the first intermediate output signal (o １ ) expressing the entropy or cross entropy of - determining an output signal (y) from the machine learning system (60) based on the image (x) and the adapted parameters (Φ); A method comprising:

2. The loss function represents the average entropy of classification obtained for multiple pixels of the image (x). The method of claim 1.

3. a second intermediate output signal is determined based on the first intermediate output signal by a transformation function, the second intermediate output signal representing a semantic segmentation and / or an instance segmentation of the image, and the cross entropy is determined based on the first intermediate output signal and the second intermediate output signal; The method of claim 1.

4. The transformation function represents an edge-preserving smoothing filter. The method of claim 3.

5. the loss function further represents a likelihood, preferably a log-likelihood, of at least a portion of the first intermediate output signal, the likelihood being determined based on a density model of the at least a portion of the image. The method of claim 1.

6. the likelihood represents an average likelihood of a plurality of patches, the plurality of patches being determined based on the first intermediate output signal. The method of claim 5.

7. the likelihood for a patch is determined by determining a feature representation for the patch using a feature extractor and providing the likelihood of the feature representation as the likelihood of the patch, the likelihood of the feature representation being determined by the density model; The method of claim 6.

8. The density model is represented by a mixture model, in particular a Gaussian mixture model, or a normal distribution. The method of claim 5.

9. To determine the output signal, the machine learning system (60) includes a normalization transform, and the loss further represents a Kullback-Leibler divergence between the output of the normalization transform and a predefined probability distribution. The method of claim 1.

10. The predefined probability distribution is represented by a standard normal distribution.

10. The method of claim 9.

11. the machine learning system (60) includes a neural network for determining the output signal (y), the neural network including a normalization layer, and the loss further representing a Kullback-Leibler divergence between an output of the normalization layer and a predefined probability distribution; 10. The method of claim 9.

12. A machine learning system (60) configured to implement the method of any one of claims 1 to 11.

13. A control system (40) for determining a control signal (A), comprising: the control signal is configured to control an actuator (10) and / or a display (10a); The control signal (A) is determined based on an output signal (y) determined according to a method according to any one of claims 1 to 11. A control system (40).

14. A computer program configured to cause a computer to carry out all the steps of the method according to any one of claims 1 to 11 when said computer program is executed by a processor (45).

15. A machine-readable storage medium (46) having stored thereon a computer program according to claim 14.

Images