Apparatus and method for amplifying feature maps of an image

Abstract

An apparatus and method for upscaling feature maps of images are provided. A computer-implemented method for training a first machine learning system comprises the steps of: training the first machine learning system in a first stage, the first stage comprising a pre-training of the first machine learning system; training the first machine learning system in a second stage, the second stage comprising: training a second machine learning system based on the first stage; obtaining a training image and a feature map of the training image; determining a first upsampled feature map by providing the training image and the feature map as input to the first machine learning system; determining a second upsampled feature map by providing a crop of the training image and a corresponding crop of the feature map as input to the second machine learning system; training the first machine learning system based on a second loss function, the second loss function characterizing a difference between a crop of the first upsampled feature map and the second upsampled feature map, the crop of the first upsampled feature map corresponding to the crop of the training image.

Description

Existing technology

[0001] Suri et al.'s "LiFT: A Surprisingly Simple Lightweight Feature Transform for Dense ViT Descriptors" (https: / / arxiv.org / abs / 2403.14625, 2024) discloses a self-supervised approach to enhance the performance of ViT features for dense downstream tasks.

[0002] Kirillov et al.’s “Segment Anything” (in ICCV, pp. 4015-4026, 2023) published a model for segmenting everything. Background Technology

[0003] Modern computer vision systems, used to process images recorded from physical reality, particularly utilize neural networks, such as, for example, visual foundation models (VFM), to extract meaningful representations from images. These representations serve as the basis for a wide range of downstream tasks, including object detection, semantic segmentation, depth estimation, and visual question answering. In these cases, neural networks such as VFM typically operate on significantly downsampled feature maps due to computational constraints and inherent architectural choices in their design (e.g., pooling layers). This reduced spatial resolution limits the accuracy of pixel-level analysis, hindering performance for tasks requiring fine-grained detail. Such tasks are particularly critical in applications involving interaction with the physical world, such as robotic manipulation, autonomous navigation, and augmented reality.

[0004] Existing solutions for overcoming this resolution bottleneck exhibit significant drawbacks. Using larger input images quadratically increases computational costs, making real-time performance challenging, especially in resource-constrained environments. Training task-specific decoders requires substantial resources and lacks generality, necessitating retraining for each new task. Neither approach provides a scalable and efficient way to extract high-resolution features directly from pre-trained neural networks such as VFM.

[0005] Furthermore, existing task-agnostic feature upsampling methods struggle to generate high-quality, high-resolution features that faithfully preserve the detailed information present in the original image. The challenge lies in designing effective training objectives and upsampling architectures capable of capturing fine-grained details without introducing artifacts that could negatively impact downstream performance. This remains a significant challenge for computer vision systems operating in real-world scenarios, where accurate pixel-level understanding is crucial for robust and reliable performance.

[0006] This invention addresses these limitations by introducing a novel method for upsampling feature maps in an unsupervised manner. It offers several key advantages: First, it enables full-resolution supervision during training, resulting in significant improvements in detail and reduced artifacts in the upsampled features. Second, it utilizes a task-agnostic training strategy, allowing the resulting upsampler to generalize to a wide variety of downstream tasks without requiring retraining. Finally, compared to known methods, this invention demonstrates superior performance across a range of downstream tasks, highlighting its effectiveness and potential for improving real-world computer vision applications. Summary of the Invention

[0007] A computer implementation method for training a first machine learning system, wherein the first machine learning system is configured to determine an upsampled output representing the feature map of the image using an image and a feature map of the image as input, the method comprising the following steps: • The first machine learning system is trained in the first phase, which includes: • Obtain the training images and their feature maps; • Obtain the desired upsampled feature map, wherein the desired feature map has a larger size than the feature map, and wherein obtaining the feature map includes interpolating the feature map; • A first machine learning system is trained based on a first loss function, wherein the first loss function characterizes the difference between the predicted upsampled feature map and the expected upsampled feature map, wherein the predicted upsampled feature map is obtained from the machine learning system as the output using the training image and feature map as input; • The first machine learning system is trained in the second phase, which includes: • Train the second machine learning system based on the first stage; • Obtain the training images and their feature maps; • The first upsampled feature map is determined by feeding the training image and feature map as input to the first machine learning system; • The second upsampled feature map is determined by feeding the cropped training image and the corresponding cropped feature map as input to the second machine learning system; • The first machine learning system is trained based on a second loss function, wherein the second loss function characterizes the difference between the cropping of the first upsampled feature map and the second upsampled feature map, and the cropping of the first upsampled feature map corresponds to the cropping of the training image.

[0008] This method can generally be understood as first performing a form of pre-training on a first machine learning system in a first stage, where the second stage can be understood as a form of self-distillation. Using common terminology in distillation training methods, the second neural network can be understood as the "teacher," and the first neural network as the "student."

[0009] The term "feature map" can be understood as a multidimensional data representation extracted from an image by a feature extractor, such as a convolutional neural network or a visual transformer. It can represent different levels of abstraction, from low-level features like edges and textures to high-level semantic features representing objects or concepts. A feature map can be a tensor, where each element can correspond to a spatial location in the image (i.e., pixel coordinates), and each channel can represent a different learned feature. In particular, a feature map can be represented as a three-dimensional tensor, where the first and second dimensions define the spatial locations of feature vectors within the feature map (i.e., spatial dimensions), while the third dimension defines the number of features for each feature vector. Due to pooling or striding operations in the feature extractor, the spatial dimension of the feature map may be smaller than that of the original image. Furthermore, a feature map can be an intermediate representation within a larger neural network, or it can be the final output of the feature extraction process. It can encode information about the image content in a compressed and computationally efficient format, suitable for use in downstream tasks such as image classification, object detection, or semantic segmentation. Feature maps can also be referred to as feature representations, activation maps, or tensor representations.

[0010] The term "image" can be broadly understood as a visual representation of a scene or object, and is not limited to traditional photographs or RGB images. An image can be a two-dimensional or multi-dimensional data array, where each element can represent a measurement or intensity at a specific spatial location. Images can be acquired from various sensors, including but not limited to cameras, lidar sensors, radar sensors, ultrasonic sensors, or other imaging devices. Images can represent different modalities, such as intensity, depth, reflectivity, or other physical properties. Specifically, as part of the claimed method, the image can be recorded by a sensor, and the data representing the image can be subsequently processed. The image can be a single frame or a portion of a series of frames, as in video. Furthermore, the image can be preprocessed, for example, for noise reduction or color correction, before being used in the method.

[0011] The expression "upsampling a feature map" can be understood as the process of increasing the spatial resolution of the feature map. This may involve interpolation between existing feature values, transformations to learn a higher-resolution representation, or a combination of both. The goal of upsampling may be to obtain a feature map with a spatial resolution closer to that of the original image, thereby enabling finer-grained analysis and better performance in tasks requiring pixel-level precision. Various techniques can be used to perform the upsampling process, such as nearest-neighbor interpolation, bilinear interpolation, bicubic interpolation, transposed convolution, or dedicated upsampling networks. In particular, upsampled feature maps can retain the same number of channels as the original feature map.

[0012] The term "feature extractor" can be understood as a component of a computer vision system, such as a neural network, that transforms an input image into a feature map. This transformation may involve extracting relevant information from the image, such as edges, textures, shapes, or higher-level semantic concepts. Feature extractors can be pre-trained on large image datasets, allowing them to learn general features useful for a wide variety of downstream tasks. Examples of feature extractors can include convolutional neural networks (CNNs), visual transformers (ViTs), or other architectures designed for image analysis. The output of the feature extractor, the feature map, can be a lower-dimensional representation of the input image, thus encoding relevant information in a more compact and computationally efficient format. The feature map can then be used as input to other components of the computer vision system, such as classifiers, detectors, or segmentation networks. Feature extractors can be standalone modules or they can be integrated into larger neural network architectures.

[0013] The inventors discovered that using a DINOv2 neural network as a feature extractor in the corresponding method of the present invention allows for more accurate upsampling of feature maps, particularly the DINOv2-S / 14 model. However, other architectures such as CLIP and RADIO can also be used.

[0014] The phrase "cropping of the training image" can be understood as a smaller rectangular portion of the training image. This cropping can be achieved using various techniques, such as random cropping, where the location and / or size of the crop is randomly selected within the image, or center cropping, where the crop is taken from the center of the image. The size of the crop can be fixed, or it can be randomly selected within a predefined range. The location and size defined by the crop can also be "applied" to the feature map of the training image by selecting feature vectors from the training image that fall within the region characterized by the same location and size as the crop in the feature map.

[0015] The phrase "the crop of the first upsampled feature map corresponds to the crop of the training image" can be understood as recognizing a portion of the first upsampled feature map that is spatially aligned with the crop obtained from the original training image. The crop extracted from a specific location and size within the training image corresponds to a specific region in the feature space of the upsampled feature map. This corresponding region can be determined by considering the scaling factor introduced during the downsampling and upsampling processes. Specifically, the crop coordinates within the original training image can be scaled according to the downsampling and upsampling factors to determine the corresponding coordinates within the upsampled feature map. The region in the upsampled feature map can have the same width and height as the crop, or it can have a different size determined by the applied scaling factor.

[0016] The phrase "training a machine learning system based on a loss function" can be understood as training a machine learning system according to common optimization strategies that employ a loss function, such as (stochastic) gradient descent with respect to the loss function of the weights (i.e., trainable parameters) of a neural network, or an evolutionary optimization strategy that uses the loss function as the objective to be minimized.

[0017] The inventors found that, advantageously, this method allows for the training of a first machine learning system with an emphasis on clear boundaries and geometric awareness, which in turn leads to improved predictions of upsampled feature maps. This is an advantage in addition to the fact that the training method can operate unsupervised (i.e., without requiring annotation of the training images).

[0018] In a preferred embodiment, the second stage may include the following steps: • Train the second machine learning system based on the exponential moving average of the parameters of the first machine learning system.

[0019] The inventors discovered that this method even further improved the prediction accuracy of the first neural network.

[0020] In a preferred embodiment, determining the desired upsampled feature map may include the following steps: • Use bicubic interpolation to generate intermediate upsampled feature maps from the feature maps; • Obtain category-agnostic masks for the training images; • The desired upsampled feature map is determined by refining the intermediate upsampled feature map based on the class-agnostic mask by mixing the average feature value of the region within the mask with the corresponding value in the intermediate upsampled feature map.

[0021] The statement "generating an intermediate upsampled feature map from a feature map using bicubic interpolation" can be understood as increasing the spatial resolution of the feature map using a bicubic interpolation algorithm. Bicubic interpolation estimates the value of a new "pixel" in the upsampled feature map by considering a weighted average of the nearest-neighbor pixels in the original feature map. The weights in this average can be determined by a bicubic function, which provides a smoother interpolation compared to simpler methods such as bilinear interpolation. Specifically, the bicubic function can be applied to both the horizontal and vertical dimensions of the feature map. This bicubic interpolation process can be applied independently to each channel of the feature map. The result of this operation can be an intermediate upsampled feature map with increased spatial dimensionality and smoother interpolation.

[0022] The expression "obtaining a category-agnostic mask for the training image" can be understood as obtaining a mask that segments the training image into different regions without assigning specific category labels to those regions. This mask can identify salient regions or objects within the image, but it does not classify them into predefined categories. Various techniques, such as unsupervised segmentation algorithms, edge detection methods, or attention mechanisms, can be used to generate category-agnostic masks. Preferably, a segmentation model such as Segment All Model (SAM) can be used to generate such a mask. The mask can be a binary image where each pixel is assigned a value of 0 or 1, indicating whether it belongs to a salient region. Alternatively, the mask can have multiple channels, where each channel can represent a different segment or region.

[0023] The statement "mixing the average feature values ​​of the regions within the mask with the corresponding values ​​in the intermediate upsampled feature map" can be understood as the process of refining the intermediate upsampled feature map using information derived from the class-agnostic mask. The mask initially obtained for the training image can be superimposed on the intermediate upsampled feature map to define the corresponding regions within it (if the mask is larger or smaller than the training image, it may be rescaled to the size of the intermediate upsampled feature map). For each connected region defined by the mask, an average feature vector can be computed. Specifically, the pixels within each region of the intermediate feature map can be treated as vectors, and the average of these vectors can be calculated to obtain the average feature vector for that region. This average feature vector can represent the average feature representation within the corresponding masked region. These average feature values ​​can then be mixed with the original values ​​in the intermediate upsampled feature map. The mixing operation can be a weighted average, where weights determine the contribution of the average feature values ​​and the original feature map values ​​to the final refined feature map. In addition to “smoothing” feature vectors that are semantically similar or identical pixels, this blending process also enhances the representation of salient regions in the refined upsampled feature map by introducing a sense of “similarity” to the feature vectors (pixels of the feature map) determined for connected regions in the training image.

[0024] Specifically, mixing can be achieved using the following formula: in, It is the first intermediate upsampled feature map obtained using bicubic interpolation. Each region It can be understood as a "mixing factor" of the hyperparameters of the training method, and It is the first each region The average value.

[0025] In a preferred embodiment, the second loss function can characterize the affinity matrix loss function.

[0026] In other words, the loss function can be any form of affinity loss, i.e., scaled affinity loss, etc. The inventors found that, advantageously, using affinity matrix loss further increased the prediction accuracy of the first neural network.

[0027] A computer implementation method for training a third machine learning system, wherein the third machine learning system is configured to accept an image as input, provide semantic segmentation of the image as output, and / or provide a depth map of the image as output, and / or provide normals of the image as output, and / or provide detected objects in the image as output, wherein the third machine learning system includes a first machine learning system according to any one of the preceding claims and a first head for providing semantic segmentation and / or a second head for providing a depth map and / or a third head for providing normals and / or a fourth head for providing detected objects, wherein the method includes the following steps: • The first machine learning system is trained according to any one of the preceding claims; • Train the first and / or second and / or third and / or fourth head.

[0028] The terms "machine learning system accepts an image as input" or "machine learning system uses an image as input" can generally be understood as the corresponding machine learning system being provided with data that makes up an image (e.g., pixels) as input and determining its output based on that data (even if the output is based on the image data itself).

[0029] The term "head" can be understood as a machine learning system, particularly a neural network, that performs a specific task in computer vision. A head can be a relatively small neural network (as small as a single layer) that takes feature maps as input and produces a desired output, such as semantic segmentation, depth maps, surface normals, or object detection. Heads can be trained independently or as part of a larger multi-task network. In the context of a multi-task network, multiple different heads can share upsampled feature maps as input, each head specializing in a different task. This allows the network to provide shared representations from the input image that can be used for multiple purposes, thereby improving efficiency and potentially enhancing performance. Depending on the specific task the head performs (e.g., object detection, semantic segmentation, video object segmentation, depth estimation, or normal estimation), a head can include various layers, such as convolutional layers, fully connected layers, or other specialized processing units. A head can be a standalone machine learning system taking feature maps as input, or it can be a component integrated into a larger neural network architecture as part of a multi-task machine learning system.

[0030] The authors found that, advantageously, the overall prediction accuracy of the above task is improved when the upsampled feature map determined as described above is used as the input to the head.

[0031] In a preferred embodiment, it is possible that the parameters of the first machine learning system are not adapted during the training of the first head and / or the second head and / or the third head and / or the fourth head.

[0032] The first machine learning system not being adapted can be understood as the first machine learning system being frozen during the training of the head.

[0033] Advantageously, the authors found that the performance of the first machine learning system in predicting accurate upsampled feature maps was good enough that it was unnecessary to train the first machine learning system for the corresponding head task. This resulted in a reduction in computational resources, as only the head needed to be considered for training.

[0034] In a preferred embodiment, the third machine learning system may further include a feature detector configured to determine feature maps from the image.

[0035] The methods proposed above can be applied to train known machine learning systems for upsampling feature maps, such as the machine learning models used in LiFT, FeatUp, Index Networks, SAPA, LIIF, and CARAFE.

[0036] However, in a preferred embodiment of the method presented above, determining the output of the first machine learning system or the second machine learning system may include: • Enhance the input image by using the corresponding representation of the pixel coordinates of the image, thereby creating an enhanced image; • Feeding the augmented image as input to the neural network includes via a first machine learning system or via a second machine learning system, wherein the neural network includes at least one cross-attention layer that uses the augmented image or its representation as query input and uses feature maps as key input; • Provide the output of the neural network as an upsampled feature map, where the output of the neural network is based on the output of the cross-attention layer.

[0037] The expression "enhance an image with corresponding pixel coordinates" can be understood as adding spatial information to the image. This may involve creating additional channels within the image representation, where the value of each pixel in these new channels corresponds to its x and y coordinates within the image. The coordinates can be represented as the original pixel index, normalized values ​​between 0 and 1, or they can be transformed using a function such as sine coding. This enhancement can provide explicit spatial cues to neural networks, allowing them to better understand the geometric relationships between different parts of the image. The enhanced image can be used as input to a neural network, or it can undergo further processing before being used.

[0038] The concept of a "cross-attention layer that uses an augmented image or its representation as a query input and a feature map as a key input" can be understood as a component of a neural network that allows interaction between two distinct feature representations (i.e., the augmented image and the feature map). The augmented image, or alternatively its representation, obtained by transforming the image within the network (e.g., through layer forwarding), serves as the query, which can be understood as a high-resolution spatial representation seeking relevant information. The feature map serves as the key, which can be understood as a lower-resolution representation containing semantic information. The cross-attention mechanism computes attention weights based on the similarity between the query (augmented image) and key (feature map) features. This similarity measure can be implemented, for example, as a dot product between the query and key vectors, followed by a scaling operation and a softmax function to normalize the weights. These attention weights are then used to selectively aggregate information from a value input, which can be the feature map itself or another relevant representation. This allows the network to combine detailed spatial information from the augmented image with richer semantic information from the feature map. Cross-attention layers can be implemented using various techniques, such as scaled dot product attention. The output of the cross-attention layer can be a novel feature representation that integrates information from both inputs, which can be understood as a high-resolution feature map rich in semantic context. This combined representation can be further processed by subsequent layers in the network.

[0039] Feature maps can also be used, in particular, as “value inputs” for cross-attention layers.

[0040] Neural networks can use a combined representation from a cross-attention layer, which integrates features from the augmented image and other sources. Figure 2 The information from the network is used to generate a magnified feature map as output in several ways; that is, the output of the neural network can be based on the output of the cross-attention layer in various ways. For example, the ensemble representation can be passed through one or more convolutional layers. These convolutional layers can learn filters that effectively combine spatial and semantic information to produce the desired high-resolution output. Alternatively, the ensemble representation can be fed into a series of upsampling layers, such as transposed convolutions or dedicated upsampling blocks. These upsampling layers can increase the spatial resolution of the feature map while preserving the learned features. Furthermore, the network can use a combination of convolutional layers and upsampling layers to refine the ensemble representation and generate the final magnified feature map.

[0041] In a preferred embodiment, the output of the cross-attention layer can form part of a transformer block, which preferably includes a feedforward network, layer normalization, and / or residual connections. The output of this transformer block can then be directly used as the output of the neural network, i.e., an upsampled feature map, thereby efficiently combining spatial and semantic information in a single processing step. The specific architecture and configuration of the transformer block following the cross-attention layer can be determined using techniques such as hyperparameter optimization or neural architecture search. Additionally, the network can incorporate skip connections, residual connections, or other architectural elements to facilitate training and improve performance.

[0042] Advantageously, the inventors found that the proposed method allows for the acquisition of upsampled feature maps that are more accurate compared to other upsampling methods. Content-aware upsampling is achieved using a cross-attention layer—where the augmented image is used as the query and the feature map as the key—resulting in a reduction of artifacts and blurring common in other methods. This leads to “sharper” and more accurate high-resolution feature maps. Furthermore, this method offers increased computational efficiency compared to implicit upsampling techniques that require iterative optimization. By leveraging a feedforward neural network, the upsampling process is significantly faster, a clear technical advantage, especially in resource-constrained environments or real-time applications. Moreover, including explicit pixel coordinates allows the method to handle arbitrary output resolutions, providing a flexibility not easily achieved by conventional upsampling techniques, which are typically limited to fixed integer scaling factors. This adaptability is a technical feature that extends the applicability of the method. The cross-attention mechanism also facilitates the efficient integration of spatial information from the augmented image with semantic information encoded in the feature map. This information fusion produces high-resolution feature maps that are both spatially detailed and semantically rich, a technical improvement that helps enhance the performance of downstream tasks requiring precise pixel-level understanding. While the improved performance in applications such as semantic segmentation, depth estimation, and object detection is an indirect benefit, it stems directly from the aforementioned technical improvements in image quality, computational efficiency, and resolution flexibility provided by this method.

[0043] In a preferred embodiment, the neural network may include or consist of multiple blocks, wherein each block includes or consists of a cross-attention layer that uses feature maps as key inputs, and wherein the output of the cross-attention layer is used as input to the fully connected neural network.

[0044] The term "multiple blocks" can be understood as a collection of two or more processing units or blocks arranged within a neural network, which can be thought of as a logical cluster of layers within the neural network. Each block—also called a "transformer block"—can perform a specific operation or a series of operations on the input data. These blocks can be identical in structure and function, or they can be different, each block dedicated to a specific aspect of the overall processing task. Specifically, each block may include a cross-attention layer that uses a feature map as a key input and an augmented image or its representation as a query input, as described above. Blocks can be arranged sequentially, with the output of one block serving as the input to the next block, or they can be arranged in more complex configurations, such as parallel or branching structures. Blocks can share parameters, or they can have independent sets of parameters. Multiple blocks can contribute to the overall depth and complexity of the neural network, enabling it to learn more complex representations and perform more complex tasks. The number of multiple blocks can vary depending on the specific application and the desired performance characteristics. The actual number of blocks can be thought of as a hyperparameter of the method and can be "tuned" using common hyperparameter optimization techniques. Blocks can be implemented using various types of layers, such as convolutional layers, fully connected layers, attention layers, or other specialized processing units.

[0045] Specifically, the neural network can consist of multiple transformer blocks, each of which consists of a cross-attention layer with inputs as described above and a fully connected neural network that uses the output of the cross-attention layer as input to determine the output of the transformer block.

[0046] Advantageously, the inventors have found that the architecture according to these preferred embodiments allows for highly accurate predictions of upsampled feature maps, while being computationally "inexpensive" because the architecture does not require iterative fine-tuning.

[0047] In a preferred embodiment, the neural network may include a convolutional layer as the first layer of the neural network, wherein the convolutional layer preserves the width and height of the image.

[0048] These preferred embodiments can be understood as neural networks employing convolutional layers that retain width and height as input layers, wherein the output of the convolutional layers can then be forwarded to other layers or portions of the neural network, particularly to multiple transformer blocks, as described in any of the foregoing embodiments.

[0049] The phrase "convolutional layers preserve the width and height of an image," as used in this article, can be understood as meaning that the spatial dimensions of the output of a convolutional layer are the same as those of the input image. This can be achieved through several techniques. For example, padding can be applied to the input image before performing convolution. This padding might add extra rows and columns of pixels around the image's boundaries, effectively increasing its size. A convolution operation can then be performed on this padded image. By choosing an appropriate padding scheme, such as "same" padding, the output of the convolution can have the same width and height as the original unpadded image. Alternatively, a convolutional layer with a stride of one and a kernel size that maintains the spatial dimensions can be used. Furthermore, extended convolutions, which introduce gaps between the elements of the convolutional kernel, can also be used to preserve the width and height of the image while increasing the receptive field of the convolutional layer.

[0050] Specifically, the neural network can consist of a 3x3 convolutional layer with 1-pixel padding as the first layer, followed by multiple transformer blocks, each of which consists of a cross-attention layer with inputs as described above and a fully connected neural network that uses the output of the cross-attention layer as input to determine the output of the transformer block.

[0051] The inventors discovered that, advantageously, using a convolutional layer as the first layer appropriately combines information from both pixel and positional codes in the image, ensuring that subsequent layers or blocks of the neural network are not biased by pixel and positional codes with different "channels" in the neural network's input. This produces a combined representation of both pixels and positions, allowing for unbiased further processing. In particular, the convolutional layer further improves the predictive accuracy of the neural network.

[0052] In a preferred embodiment, the coordinates of the pixels in the image can be represented by the sinusoidal position encoding of the corresponding positions of the pixels within the image.

[0053] The term "sine positional coding" can be understood as encoding the position of pixels within an image using a sine function. This coding can be achieved by generating a set of sine and cosine waves with different frequencies, where each frequency can correspond to a different spatial dimension (e.g., x and y coordinates). For each pixel, the values ​​of these sine and cosine waves at the pixel coordinates can be calculated and concatenated to form a positional code. This sinusoidal positional code can then be combined with the pixel values ​​of the image, for example, through concatenation, to create an enhanced image representation.

[0054] Advantageously, the use of sine functions can provide a smooth and continuous representation of positions, allowing neural networks to learn spatial relationships between pixels more effectively. Furthermore, sine coding enables the network to generalize to images of different sizes, as the positional encoding can be scaled or extrapolated to fit different image dimensions. These aspects lead to further improvements in the prediction accuracy of neural networks.

[0055] In another aspect, the present invention relates to a training system configured to implement the training method.

[0056] In another aspect, the present invention relates to a control system configured to use a third machine learning system to determine control signals, wherein the control signals are configured to control actuators and / or displays.

[0057] In another aspect, the present invention relates to a computer program configured to cause a computer to perform any of the methods and all of the steps thereof when the computer program is executed by a processor.

[0058] In another aspect, the present invention relates to a machine-readable storage medium on which a computer program is stored. Attached Figure Description

[0059] Embodiments of the invention will be discussed in more detail with reference to the following figures. The figures illustrate: Figure 1 The first machine learning system is configured to upsample the feature map; Figure 2 The first phase of the method used to train the first machine learning system; Figure 3 The second phase of the method used to train the first machine learning system; Figure 4 The training system used to train the first machine learning system; Figure 5 The third machine learning system is configured to predict different outputs; Figure 6 A third machine learning system is used to determine the control signals of the control system; Figure 7 A control system that controls at least partially automated robots; Figure 8 Control systems for manufacturing machines. Detailed Implementation

[0060] Figure 1 A first machine learning system (60) is schematically shown, which is configured to run on the provided image ( The feature map provided by ) The method of upsampling. Image ( The input is provided to the machine learning system (60). Then, the pixel coordinates are used as the input. ) for image ( Enhancement is performed to determine the enhanced image. In this embodiment, two-dimensional sine coding is used as the representation (). However, other representations are also possible in the embodiments. Representation ( The image has the same width and height dimensions as the image. To determine the enhanced image ( ),image( ) and represent ( Therefore, it can exist in the form of a three-dimensional tensor, where enhancement is achieved by connecting the image tensor and the representation tensor along a dimension orthogonal to the width and height dimensions of the respective tensor.

[0061] Enhance the image ( The neural network (61) is provided as input to the machine learning system (60). The neural network (61) includes optional convolutional layers (62) as the first layer, specifically using the "same" padding strategy and including 3x3 convolutional layers with fewer than 1024 filters, specifically fewer than 512 filters, specifically 384 filters. The output of the convolutional layers can be understood as an enhanced representation of the image. ), especially images ( The representation of () ).express( The input is provided to the cross-attention layer (CA) as query input. Feature maps are also provided to the cross-attention layer (CA). ( ) is used as a key input. Preferably, a feature map is also provided to the cross-attention layer (CA). ) as the value input.

[0062] The output of the cross-attention layer (which can again be understood as a representation of the image) Although numerically different from the representation determined from the convolutional layer ( The input is preferably provided to a fully connected neural network (FCNN), particularly a multilayer perceptron. The multilayer perceptron can be a three-layer perceptron, particularly with each layer using the same number of neurons, where the total number of parameters of the multilayer perceptron is the square of the number of filters used in the convolutional layers, or, in the absence of convolutional layers, the total number of parameters of the multilayer perceptron is 147456. The fully connected neural network (FCNN) determines another representation of the image (…). The cross-attention layer and the fully connected neural network (FCNN) can be understood as "logical" blocks belonging to the neural network. ), especially the converter block ( The output of the block (i.e., the output of the fully connected neural network (FCNN)) can be provided as the output of the neural network (61). Alternatively, the neural network (61) may include multiple blocks ( ), of which blocks ( ) sequential connection. Block ( The number of blocks (59) can be considered a hyperparameter of the machine learning system. Preferably, the neural network (61) comprises fewer than 20 blocks. ), especially fewer than 10 blocks ( ), especially the 7 blocks ( ).

[0063] Figure 2 This illustrates the first stage of a method for training a first machine learning system. Training images ( ) is provided to the feature extractor (63) for determining the feature map ( Then, the feature map ( The image and the input are provided to the first machine learning system (60). The first machine learning system (60) determines the upsampled feature map based on the input. ) as output.

[0064] Training images ( The class-agnostic segmentation module (66) is also provided to the class-agnostic segmentation module (66). The class-agnostic segmentation module (66) is configured to determine the training image ( Category-unknown mask () ), especially based on training images ( The semantic segmentation map is used to segment the training image. The corresponding part of ). The category-agnostic segmentation module (66) can in particular employ SAM, that is, segmenting all models to determine the category-agnostic mask ( ).

[0065] Feature map ( The feature map (64) is also provided to the upsampling module. The upsampling module is configured to upsample the feature map (64). Upsampling is performed, specifically upsampling to the training image ( The size of the upsampling module (64) can be determined, in particular, by using bicubic upsampling to determine the intermediate upsampling feature map. Intermediate upsampled feature map () ) and category-agnostic mask ( ) is provided as input to the mixing module (65), wherein the mixing module is configured to upsample the intermediate feature map ( The feature vectors of the image are mixed with the corresponding means of the regions to which the feature vectors belong. Initially, the training image ( The mask obtained () ) can be superimposed on the intermediate upsampled feature map ( On ), to limit the intermediate upsampled feature map ( The corresponding region within the mask (). For each connected region defined by , an average feature vector can be calculated. Specifically, the intermediate upsampled feature map ( Each "pixel" within a region can be treated as a vector, and the average of these vectors can be calculated to obtain the average feature vector for that region. This average feature vector can represent the average feature representation within the corresponding masked region. These average feature values ​​can then be compared with the intermediate upsampled feature map (…). The original values ​​in the feature map are mixed. The mixing operation can be a weighted average, where the weights determine the contribution of the average feature value and the original feature map value to the final refined feature map.

[0066] Specifically, mixing can be achieved using the following formula: in, It is an intermediate upsampled feature map ( ) Each region It can be understood as a "mixing factor" of the hyperparameters of the training method, and It is the first each region The average value.

[0067] The output of this hybrid operation is used as the desired upsampled feature map. ) provides and serves as a function in the loss function ( The input is the upsampled feature map. ) is provided as input to the loss function ( Loss function () ) is configured to determine the upsampled feature map ( ) and the expected feature map ( The deviation of the loss function ( ). In particular, the loss function ( ) It can be determined by the Euclidean loss function, Loss function or square The loss function is given. Based on the loss function ( ), can determine the loss value ( Then, the weights (i.e., trainable parameters) of the first machine learning system (60), especially the weights of the neural network (61), can be based on the loss value ( Adaptation is performed, for example, by adjusting the loss value relative to the weights of the first machine learning system (60). Automatic differentiation is performed, and the weights are updated according to the gradient. The weights of the feature extractor (63) can be specifically left unadapted, i.e., the feature extractor (63) can be frozen. A batch of training images can also be used. Training can be performed using the SA1B dataset. However, a (randomly sampled) subset of one million training images from the SA1B dataset is sufficient. Training images ( It can be specifically downsampled to a size of 224 by 224 pixels or 448 by 448 pixels for training the first machine learning system (60).

[0068] Figure 3 As shown Figure 6 The second phase of the training method presented herein. In the first step, according to Figure 6 The method is to pretrain the second machine learning system (60b) (preferably using the same architecture as the first machine learning system (60)).

[0069] Then, for the provided training images ( ), from training images ( Determine the cutting method in ) Cutting () In particular, it can have the size used for pre-training the corresponding neural network (61), for example, 224 by 224 pixels or 448 by 448 pixels. Training images ( ) can then be downsampled to a size suitable for pre-training and can be fed as input to the feature extractor (63) and as input to the first machine learning system (60), where cropping ( The first upsampled feature map can be fed as input to the feature extractor (63) (preferably the same as the feature extractor used in the first machine learning system (60)) and as input to the second machine learning system (60b). The first machine learning system (60) determines the first upsampled feature map. Based on training images ( Crop within ) The position and size of ) can be determined by analyzing the first upsampled feature map ( Extracting from the cropped area () ) relative to the training image ( The first upsampled feature map is determined by the feature vectors in the region defined by ( ). ) area ( ).

[0070] The second machine learning system (60b) determines the second upsampled feature map ( The second upsampled feature map (67) is provided as input to the downsampling module (67). The downsampling module (67) is configured to input the second upsampled feature map (67) into the downsampling module (67). Downsampling to region ( The size of the downsampled second feature map is determined by this. Then, the region ( ) and the downsampled second feature map ( ) provided to the loss function ( ), the loss function ( ) characterization region ( ) and the downsampled second feature map ( The differences between them. In particular, the loss function ( The loss function can be the affinity matrix. Determine the loss value ( ), this loss value ( Then it can be used to adapt the weights of the first machine learning system (60), such as for the weights of the first machine learning system (60). Figure 6 The second machine learning system (60b) can preferably be trained by an exponential moving average of the weights of the first machine learning system (60). Preferably, the feature extractors (63) of both the first machine learning system (60) and the second machine learning system (60b) are frozen during training.

[0071] Figure 4 An embodiment of a training system (140) for training a first machine learning system (60) using a training dataset (T) is shown. The training dataset (T) includes multiple training images for training the first machine learning system (60). ).

[0072] For training, the training data unit (150) accesses a computer-implemented database (St2) that provides a training dataset (T). The training data unit (150) preferably randomly selects at least one training image from the training dataset (T). ) and training images ( ) is transmitted to the modification unit (180).

[0073] The modification unit (180) is configured to perform Figure 2 and Figure 3 The method described in the paper is used to train the first machine learning system (60).

[0074] Modification unit (180) based on the first loss value ( ) or second loss value ( ) to determine the new parameters ( In a given embodiment, this is accomplished using a gradient descent method, preferably stochastic gradient descent, Adam, or AdamW. In another embodiment, training may also be based on an evolutionary algorithm or a second-order method used to train the neural network.

[0075] In other preferred embodiments, the described training is iteratively repeated a predefined number of times, or iteratively repeated until a first loss value falls below a predefined threshold. Alternatively or additionally, it is also conceivable that training terminates when the average first loss value on the test or validation dataset falls below a predefined threshold. In at least one iteration, new parameters determined in the previous iteration ( ) is used as a parameter of the machine learning system (60) ).

[0076] Furthermore, the training system (140) may include at least one processor (145) and at least one machine-readable storage medium (146) containing instructions which, when executed by the processor (145), cause the training system (140) to perform a training method according to one aspect of the invention.

[0077] Figure 5 A third machine learning system (70) is shown, which is configured to perform the task of determining an image ( semantic segmentation () ) and / or images ( Depth estimation () ) and / or detect images ( Objects within ) The third machine learning system (70) includes the first machine learning system (60) and a method for extracting images ( Feature map of ) The feature extractor (63) is preferably based on... Figure 2 and Figure 3 The method is used to train the first machine learning system (60). Preferably, the feature extractor (63) is a neural network, particularly a DINOv2-S14 neural network. However, other feature extractors (63) are also possible, such as CLIP neural networks or RADIO neural networks.

[0078] image( The feature map is provided as input to both the first machine learning system (60) and the feature extractor (63). The feature extractor (63) determines the feature map. ), and provide it as input to the first machine learning system (60). The first machine learning system (60) determines the upsampled feature map ( ) as output, and the upsampled feature map ( ) is provided as input to the head ( ).head( ) is configured to determine semantic segmentation ( ) or depth estimation ( ) or object detection ( Preferably, the upsampled feature map ( ) can be provided as input to multiple heads ( ), especially configured to use upsampled feature maps ( ) as a shared input representation in order to determine the corresponding output to simultaneously complete multiple different heads of multiple different tasks ( (also known as multi-task machine learning).

[0079] head( ) output ( ) or when using multiple heads ( In the case of ), head ( The output of ) is then provided as the output of the second machine learning system (58). ).

[0080] Figure 6 An embodiment is shown in which a third machine learning system (70) can be used to determine the control signal (A) of the actuator (10) in the environment (20) of the actuator (10). The actuator (10) interacts with the control system (40). The actuator (10) and its environment (20) will be collectively referred to as the actuator system. At preferably uniformly spaced time points, the sensor (30) senses the condition of the actuator system. The sensor (30) may include several sensors. Preferably, the sensor (30) is an optical sensor that captures an image of the environment (20). The output signal (S) of the sensor (30) (or, in the case where the sensor (30) includes multiple sensors, the output signal (S) of each sensor) is transmitted to the control system (40), which encodes the sensed condition.

[0081] Thus, the control system (40) receives a stream of sensor signals (S). It then calculates a series of control signals (A) based on the stream of sensor signals (S), which are then transmitted to the actuator (10).

[0082] The control system (40) receives the sensor signal (S) stream from the sensor (30) in an optional receiving unit (50). The receiving unit (50) transforms the sensor signal (S) into an input image ( Alternatively, in the absence of a receiving unit (50), each sensor signal (S) can be directly taken as the input image. Input image () For example, it can be given as an extract from the sensor signal (S). Alternatively, the sensor signal (S) can be processed to produce the input image ( In other words, the input image is provided based on the sensor signal (S). ).

[0083] Input image ( Then it is provided as input to a third machine learning system (70).

[0084] The third machine learning system (70) consists of parameters ( ) parameterization, and these parameters are stored in a parameter storage device ( ) and by parameter storage device ( )supply.

[0085] The third machine learning system (70) learns from the input signal ( Determine the output signal in ) Output signal () This includes assigning one or more labels to the input image. Information about the output signal. The signal is transmitted to an optional conversion unit (80), which outputs the signal. The signal is converted into a control signal (A). The control signal (A) is then transmitted to the actuator (10) for corresponding control of the actuator (10). Alternatively, the output signal (A) is converted into a control signal (A). ) can be directly taken as a control signal (A).

[0086] The actuator (10) receives a control signal (A), is controlled accordingly, and performs an action corresponding to the control signal (A). The actuator (10) may include control logic that transforms the control signal (A) into another control signal, which is then used to control the actuator (10).

[0087] In another embodiment, the control system (40) may include a sensor (30). In yet another embodiment, the control system (40) may alternatively or additionally include an actuator (10).

[0088] In yet another embodiment, instead of the actuator (10) or in addition to the actuator (10), a control system (40) can be envisioned to control the display (10a).

[0089] Furthermore, the control system (40) may include at least one processor (45) and at least one machine-readable storage medium (46) thereon storing instructions which, if executed, cause the control system (40) to perform a method according to one aspect of the invention.

[0090] Figure 7 An embodiment is shown in which the control system (40) is used to control at least a partially automated robot, such as at least a partially automated vehicle (100).

[0091] The sensor (30) may include one or more video sensors and / or one or more radar sensors and / or one or more ultrasonic sensors and / or one or more lidar sensors. Some or all of these sensors are preferably, but not necessarily, integrated into the vehicle (100).

[0092] The second machine learning system (70) can be configured to be based on the input image ( ) detect objects near at least a partially autonomous robot and / or input images ( The space is divided into at least drivable and non-drivable areas. Output signal ( The control signal (A) can include information indicating where the object is located near at least the partially automated robot. This information can then be used to determine control signals (A), such as to avoid collisions with the detected object while maintaining the robot within a permissible drivable area.

[0093] The actuator (10), preferably integrated into the vehicle (100), can be provided by the vehicle's (100) brakes, propulsion system, engine, transmission, or steering mechanism. A control signal (A) can be determined to control the actuator (10) so that the vehicle (100) avoids collisions with detected objects. The detected objects can also be classified according to what a second machine learning system (70) considers them to be most likely (e.g., pedestrians or trees), and the control signal (A) can be determined based on the classification.

[0094] Alternatively or additionally, the control signal (A) may also be used to control the display (10a), for example, to display objects detected by the second machine learning system (70). It is also conceivable that the control signal (A) could control the display (10a) such that if the vehicle (100) approaches and collides with at least one detected object, the display (10a) generates a warning signal. The warning signal could be an audible warning and / or tactile signal, such as vibration of the vehicle's steering wheel.

[0095] In another embodiment, the at least partially autonomous robot may be provided by another mobile robot (not shown), which may move, for example, by flying, swimming, diving, or walking. The mobile robot may, in particular, be an at least partially autonomous lawnmower or an at least partially autonomous cleaning robot. In all the above embodiments, a control signal (A) may be determined such that the propulsion unit and / or steering device and / or brakes of the mobile robot are controlled, enabling the mobile robot to avoid collisions with the identified object.

[0096] In another embodiment, at least a portion of the autonomous robot may be provided by a gardening robot (not shown) that uses sensors (30), preferably optical sensors, to determine the state of the plants in the environment (20). The actuator (10) may control nozzles and / or cutting devices (e.g., blades) for spraying liquid. Depending on the identified variety and / or identified state of the plant, a control signal (A) may be determined to cause the actuator (10) to spray and / or cut the plant with an appropriate amount of liquid.

[0097] In a further embodiment, at least part of the autonomous robot may be provided by a household appliance (not shown), such as, for example, a washing machine, stove, oven, microwave oven, or dishwasher. A sensor (30), such as an optical sensor, may detect the state of the object to be processed by the household appliance. For example, in the case of a washing machine, the sensor (30) may detect the state of the clothes inside the washing machine. The control signal (A) may then be determined based on the material of the detected clothes.

[0098] Figure 8 One embodiment is shown in which a control system (40) is used to control a manufacturing machine (11), such as a press, cutter, gun drill, or jig, of a manufacturing system (200) that is part of a production line. The manufacturing machine may include transport equipment, such as a conveyor belt or assembly line, for moving manufactured products (12). The control system (40) controls actuators (10), which in turn control the manufacturing machine (11).

[0099] The sensor (30) can be provided by an optical sensor that captures, for example, the properties of the manufactured product (12).

[0100] A third machine learning system (70) can be configured to determine the position of the manufactured product (12) relative to the transport equipment, i.e., to detect objects on or relative to the transport equipment. The actuator (10) can then be controlled, depending on the determined position of the manufactured product (12), for subsequent manufacturing steps of the manufactured product (12). For example, the actuator (10) can be controlled to cut the manufactured product at a specific location on the manufactured product itself. Alternatively, it is conceivable that the third machine learning system (70) classifies whether the manufactured product is damaged and / or exhibits defects. The actuator (10) can then be controlled to remove the manufactured product from the transport equipment.

[0101] The term "computer" can be understood as any device used to process predefined computational rules. These computational rules can be in the form of software, hardware, or a combination of both.

[0102] Typically, "multiple" can be understood as being indexed, meaning each element in the multiple is assigned a unique index, preferably by assigning consecutive integers to the elements contained in the multiple. Preferably, if the multiple includes... There are elements, among which If the number of elements is a multiple number, then the elements are assigned from... arrive An integer. It can also be understood that elements in multiple integers can be accessed through their indices.

Claims

1. A computer implementation method for training a first machine learning system, wherein the first machine learning system (60) is configured to use images ( ) and images ( Feature map of ) ) as input to determine the representation image ( The output of the upsampled feature map () The method includes the following steps: • The first machine learning system is trained in the first stage (60), which includes: • Acquire training images ( ) and training images ( Feature map of ) ); • Obtain the desired upsampled feature map ( ), where the expected feature map ( ) has a higher feature map ( A larger size, and in which obtaining the feature map includes processing the feature map ( Interpolation is performed. • Based on the first loss function ( ) train the first machine learning system (60), where the first loss function ( The upsampled feature map representing the prediction ( ) and the expected upsampled feature map ( The difference between the training images () ) and feature map ( As input, the upsampled feature map of the prediction is obtained from the machine learning system. ) as output; • The first machine learning system (60) is trained in the second stage, which includes: • Train the second machine learning system (60b) based on the first phase; • Acquire training images ( ) and training images ( Feature map of ) ); • By training images ( ) and feature map ( ) is provided as input to the first machine learning system (60) to determine the first upsampled feature map ( ); • By training images ( ) cutting ( ) and feature map ( The corresponding cropping of the feature map is provided as input to the second machine learning system (60b) to determine the second upsampled feature map; • Based on the second loss function ( ) train the first machine learning system (60), where the second loss function ( Characterizes the first upsampled feature map ( ) cutting ( ) and the second upsampled feature map ( The difference between the first upsampled feature map and the first upsampled feature map. ) cutting ( ) corresponds to the training image ( ) cutting ( ).

2. The method of claim 1, wherein the second stage comprises the following steps: • The second machine learning system (60b) is trained based on the exponential moving average of the parameters of the first machine learning system (60).

3. The method according to claim 1 or 2, wherein the desired upsampled feature map is determined ( This includes the following steps: • Use bicubic interpolation from feature maps ( Generate intermediate upsampled feature maps. ); • Acquire training images ( Category-unknown mask () ); • By masking ( The average eigenvalues ​​of the region within the range and the intermediate upsampled feature map ( The corresponding values ​​in the mixture are based on the category-agnostic mask. To refine the intermediate upsampled feature map ( ), thereby determining the desired upsampled feature map ( ).

4. The method according to any one of the preceding claims, wherein, Second loss function ( The affinity matrix loss function is used to characterize the affinity matrix.

5. A computer implementation method for training a third machine learning system (70), wherein the third machine learning system (70) is configured to receive images ( As input, semantic segmentation of the image is provided as output. ) and / or provide a depth map of the image as output ( ) and / or provide images ( The normals of the image are used as output, and / or the objects detected in the image are provided as output. The third machine learning system (70) includes a first machine learning system (60) according to any one of the preceding claims and a first head for providing semantic segmentation. ) and / or a second head used to provide a depth map ( ) and / or a third head for providing normals and / or a fourth head for providing detected objects ( The method includes the following steps: • Train the first machine learning system according to any one of the preceding claims (60); Training the first head ( ) and / or the second head ( ) and / or the third head and / or the fourth head ( ).

6. The method according to claim 4, wherein, At the first head ( ) and / or the second head ( ) and / or the third head and / or the fourth head ( During the training of the first machine learning system (60), the parameters are not adapted.

7. The method according to claim 5 or 6, wherein the third machine learning system further comprises a feature detector (63) configured to extract features from an image ( Determine the feature map in ) ).

8. The method according to any one of the preceding claims, wherein, Determining the output of the first machine learning system (60) or the second machine learning system (60b) includes: • Represented by the corresponding coordinates of the image's pixels ( To enhance the image used as input ( ), thereby creating an enhanced image ( ); • Enhance the image ( The input to the neural network (61) includes either a first machine learning system (60) or a second machine learning system (60b), wherein the neural network (61) includes at least one cross-attention layer (CA) that uses the enhanced image ( ) or its representation ( ) as query input, and use feature maps ( ) as key input; • Provides the output of the neural network (61) ) as an upsampled feature map, where the output of the neural network (61) ( Based on the output of the cross attention layer (CA).

9. The method of claim 8, wherein the neural network (61) comprises a plurality of blocks ( ) or composed of multiple blocks ( It consists of ) blocks, where each block ( This includes using feature maps ( Cross-attention (CA) layers using feature maps as key inputs or by using feature maps as key inputs. The key input consists of a cross-attention layer (CA), and the output of the cross-attention layer (CA) is... ) is used as input to a fully connected neural network (FCNN).

10. The method according to claim 8 or 9, wherein the neural network (61) includes a convolutional layer (62) as the first layer of the neural network (61), wherein the convolutional layer (62) preserves the width and height of the image.

11. The method according to any one of claims 8 to 10, wherein the image ( Representation of pixel coordinates () ) is an image ( The sinusoidal position code of the corresponding position of the pixel within () ).

12. A training system (140) configured to implement the training method according to any one of claims 1 to 11.

13. A control system (40) configured to use a third machine learning system (70) according to any one of claims 5 to 7 to determine a control signal (A), wherein the control signal (A) is configured to control an actuator (10) and / or a display (10a).

14. A computer program configured to cause a computer to perform the method and all steps thereof according to any one of claims 1 to 11, when the computer program is implemented by a processor (45, 145).

15. A machine-readable storage medium (46, 146) having thereon stored a computer program according to claim 14.

Images