Flexible segmentation of images

Abstract

A segmentation of the input image is determined.SOLUTION: In the method 100, the visual processing network 2a outputs 110 a mask 3 specifying a set of pixels belonging to different object types and an associated weight matrix 4 indicating discriminative features characterizing entities of different types in the incoming image 1, transforms 120 the weight matrix combined with the incoming image into a mask representation 6 in a latent space by an encoder network 5, and transforms the incoming image together with the mask representation by the visual processing network 2b, 2c: Processing 130 into a refinement 3 *, 3 * * and a refined weight matrix 4 *, 4 * * for the mask, transforming 140 the refined weight matrix combined with the input image into at least one refined mask representation 6 *, 6 * * in a latent space by the encoder network, and computing 150 the sought segmentation 7 from the mask and the mask representation in the latent space.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to image segmentation, which can be used in particular to process images for the purposes of autonomous driving of vehicles or robots. [Background technology]

[0002] background When a vehicle or robot is operated in an at least partially automated manner within a company premises or in public road traffic, it is necessary to constantly monitor the environment of the vehicle or robot for any objects that may be relevant for planning the future behavior of the vehicle or robot. For this purpose, images of this environment are often processed into semantic segmentations, which assign to each image pixel the class of object to which this particular pixel belongs. Summary of the Invention [Problem to be solved by the invention]

[0003] In simple applications, the set of available classes is fixed a priori, i.e., each pixel can only be attributed to a class selected from a fixed catalog. However, in autonomous driving applications, monitoring the environment must also deal with unexpected situations that contain objects that do not fit into the previously fixed catalog of classes. It is therefore desirable to make image segmentation more flexible, and in particular to perform "open vocabulary" segmentation that is not limited to such a catalog. [Means for solving the problem]

[0004] Disclosure of the Invention The present invention provides a method for determining a segmentation of an input image. The input image includes pixels having pixel values. Segmentation assigns to each pixel of the input image the class of the entity causing that pixel's pixel value. In this case, the entity may be, in particular, an object (such as a car, a pedestrian, or a tree), but may also be a non-object entity such as the sky or a road. Such non-object entities are sometimes referred to as "stuff" in the field of computer vision. The set of available classes need not be fixed a priori; that is, even if no textual meaning is attributed to any class, the segmentation is, at least in this respect, an open-vocabulary segmentation. In particular, an image may be any array of pixels, the pixel values ​​of which represent values ​​of measured data corresponding to points in space indicated by the position of each pixel in the array. For example, the image may be a still or video image recorded by a camera, a radar image, a lidar image, an ultrasound image, a thermal image, or any multimodal combination of measured data.

[0005] During the method, an input image is provided to a visual processing network. The visual processing network outputs pixel masks. These pixel masks specify sets of pixels that belong to different object types; i.e., there can be masks for different abstract object types A, B, C, D, etc., without any additional information about what exactly constitutes each type of object. In particular, the number of different masks resulting from the process need not be fixed in advance. If the visual processing network discovers that there are N classes worth distinguishing from one another, the visual processing network will output N masks. The visual processing network also outputs an associated weight matrix, which indicates discriminative features that characterize the different types of entities in the input image. Such discriminative features may relate to the presence, particularly location, of particular shapes or other distinguishable features that can be relied upon to identify the different types of entities.

[0006] In particular, the weight matrix, when applied in conjunction with a mask, may allow the mask, which is typically binary, to be softened. The weightings in the weight matrix may be tailored to subsequent processing steps, such as the use of an encoder network or other visual processing network. That is, features that downstream networks may rely on to more easily distinguish between different types of entities may be weighted more heavily.

[0007] The weight matrix need not be directly interpretable. However, the weight matrix may represent semantic interdependencies between image features and different types of objects. For example, some image features may have particular meanings or may otherwise be particularly important in relation to other features.

[0008] The encoder network converts the input image and the combined weight matrix into at least one mask representation in a latent space. This mask representation is a notion of a class assignment to the mask. The input image, together with this representation, is processed by the visual processing network into a refinement of the mask on the one hand and a refined weight matrix on the other hand. In particular, the visual processing network may be of the same architecture as the visual processing network that provided the initial estimates of the mask and the weight matrix. This means that the instance of the visual processing network used first is also of the same architecture configured to accept the mask representation as input. However, any initialization of the mask representation (e.g., random initialization) may be used to provide the initial estimates of the mask and the weight matrix. That is, the mask representation can be considered as a "learnable query" that is provided to the visual processing network in addition to the input image to extract information from the input image, and is simultaneously learned (refined) from one iteration to the next.

[0009] The encoder network then converts the refined weight matrix combined with the input image into at least one refined mask representation in latent space. At this point, the overall processing result available is of the same kind as after the first use of the encoder network: a mask on the one hand and a mask embedding on the other, but refined. This process can continue for any number of further iterations until any suitable termination criterion determines that a final sought segmentation has been computed.

[0010] That is, the final sought segmentation is computed from the initial mask and at least one representation of the initial mask in latent space, or any further refinements thereof obtained by further passes through the vision processing network and the encoder network.

[0011] The representation may have lower dimensionality than the mask itself, but may be of the same dimensionality to facilitate assembly of the final segmentation.

[0012] In one example, masks on the one hand and corresponding mask representations on the other hand can be aggregated by computing their dot product (i.e., scalar product). This results in a single numerical value, or "mask logit," that can be assigned to all pixels specified by each mask. Masks may overlap, i.e., a single and identical pixel may be specified by multiple masks. This ambiguity can be resolved by performing an argmax for each pixel, i.e., determining the mask with the highest mask logit for each pixel. The pixel can then be assigned a class corresponding to this mask.

[0013] That is, in a particularly advantageous embodiment, calculating the sought segmentation comprises calculating a dot product between at least one mask and at least one mask representation.

[0014] It has been discovered that by calculating and refining the weight matrix from one iteration to the next, the "query" provided to the visual processing network depends on the actual input image. Compared to approaches that still use such queries but limit them to a fixed number or even a fixed set, this provides significantly better flexibility. This, in turn, improves the accuracy of the segmentation ultimately obtained, both for classes seen during training of one or more visual processing networks and for classes not seen as part of the training. The instance-by-instance query used in this method allows the visual processing network to adapt to each input image for the query. This can significantly improve segmentation performance, especially in open vocabulary scenarios where the test data set may be different from the training data set and a fixed set of queries may not be optimal for use.

[0015] Furthermore, by dividing the final segmentation into multiple iterations, the overall task of obtaining the final segmentation becomes easier: a simple analogy is that walking up a flight of stairs from one floor of a building to the next higher floor is much easier than jumping directly to the next higher floor, dividing the task into many steps.

[0016] Training a network device to implement the method is further facilitated by employing an encoder network, which is not tied to this particular purpose. Rather, a generically trained encoder network can be used as is, and the parameters characterizing the behavior of the encoder network can remain frozen while one or more visual processing networks are trained.

[0017] In a particularly advantageous embodiment, an image encoder network that has been trained together with a text encoder network to estimate the best pair between image and text inputs is selected as the encoder network. For example, an image encoder network of a Contrastive Language-Image Pre-training (CLIP) network can be selected as the encoder network. By having an encoder network configured to process images but trained together with a text encoder network, semantic knowledge learned from correlations and links between text features, on the one hand, and image features, on the other, is transferred to the task of determining mask representations. For example, a CLIP network is trained using a large number of combinations of text and images, and in this process, it learns a great deal about semantic logic. If a weight matrix is ​​input to the CLIP network along with the input image, the CLIP network will apply this semantic logic when determining the next iteration of mask embedding. Furthermore, by reusing the encoder network in each iteration, the alignment between text features and image features is maintained across iterations. That is, multiple successive applications of one or more visual processing networks do not "slip up."

[0018] The use of an encoder network aligned with a text encoder network offers the additional advantage of being able to infer some notion of class meaning, facilitating further interpretation of the final segmentation map, even if some classes are not seen during training. Thus, in a further particularly advantageous embodiment, the text encoder network is used to compute text representations in latent space from candidate class names. These text representations are compared with mask representations in latent space. The results of this comparison determine the assignment of candidate class names to matching masks. Even though it is not possible to directly derive the complete text definitions of the classes, the class name assignments in this approach are highly useful for understanding the segmentation map.

[0019] In a further particularly advantageous embodiment, at least one visual processing network is a vision transformer network that calculates attention relations between portions of its input. These attention relations preserve much of the logic relied upon to synthesize the input image. In particular, the input images generated in an automated driving task are not random composites of objects. Rather, the input images generated in an automated driving task are images of scenes in the vehicle's or robot's environment that obey at least some logic, i.e., basic physical laws (e.g., gravity exists and objects do not float freely in space) and basic traffic rules. Another advantage of transformer networks is that they can easily accept additional inputs on top of the image, i.e., mask representations.

[0020] In particular, the weight matrix can be computed from the attention relations, e.g., the attention biases output by the vision transformer network can be used directly.

[0021] In a further particularly advantageous embodiment, the encoder network is selected to be a further transformer network that calculates attention relationships between portions of its input. The weight matrix can then be directly applied to at least one attention layer of this further transformer network. In particular, the attention bias calculated by the vision transformer network can be applied to at least one attention layer of the further transformer network.

[0022] Any refinements to the mask can advantageously be calculated as offsets to be applied to the initially calculated mask, in this way the amount to be sought is less than if an entire new mask had to be determined.

[0023] Each instance of the visual processing network may be used and trained separately, providing maximum flexibility at the expense of increasing the overall size of the network device. If the size of the network is to be reduced, a particularly advantageous embodiment may use a single and identical visual processing network for the initial calculation of the mask and weight matrix, on the one hand, and for the refinement of the mask and weight matrix, on the other hand. The function that the visual processing network should perform in each application may then be controlled by additional inputs to this visual processing network. For example, the visual processing network may receive a stage indicator s as an additional input.

[0024] Advantageously, the input image may be an image acquired by at least one sensor. Then, an action signal can be calculated from the segmentation obtained from this input image as described above. A vehicle, a robot, a driver assistance system, a quality inspection system, a surveillance system, and / or a medical imaging system can then be operated by the action signal. Even in the presence of objects of classes not seen during training, the improved accuracy of the segmentation increases the probability that the reaction performed by the respective operated system in response to the action signal will be appropriate in the situation characterized by the input image. In particular, unexpected situations that are more likely to give rise to the appearance of unseen classes can be better handled.

[0025] The method may be wholly or partly computer-implemented and may be implemented in software. Accordingly, the present invention also relates to a computer program comprising machine-readable instructions that, when executed by one or more computers and / or computing instances, cause the one or more computers and / or computing instances to implement the above-described method. In this specification, control units for vehicles or robots and for other embedded systems capable of executing machine-readable instructions should also be considered as computers. Computing instances include virtual machines, containers, or other execution environments that enable the execution of machine-readable instructions in the cloud.

[0026] The non-transitory storage medium and / or downloadable product may include a computer program. A downloadable product is an electronic product that can be sold online and transferred over a network for immediate fulfillment. One or more computers and / or computing instances may be equipped with such computer program and / or such non-transitory storage medium and / or downloadable product.

[0027] The present invention will now be described with reference to the drawings, without any intention of limiting the scope of the invention. [Brief explanation of the drawings]

[0028] [Figure 1] 1 illustrates an exemplary embodiment of a method 100 for determining a segmentation 7 of an input image 1. FIG. [Figure 2] FIG. 1 illustrates exemplary processing of an input image 1 in multiple instances 2a-2c of a vision transformer. [Figure 3] 1 illustrates an exemplary interaction between a mask 3 and a corresponding mask embedding 6 in the computation of a segmentation 7. FIG. [Figure 4] FIG. 10 illustrates an exemplary assignment of class names 9, 9' to masks 3, 3'. DETAILED DESCRIPTION OF THE INVENTION

[0029] 1 is a schematic flow chart of an exemplary embodiment of a method 100 for determining a segmentation 7 of an input image 1. The input image 1 comprises pixels having pixel values. The segmentation 7 assigns to each pixel of the input image 1 a class of the entity causing the pixel value of that pixel.

[0030] According to block 105, an input image 1 acquired by at least one sensor can be selected.

[0031] In step 110, an input image 1 is provided to a visual processing network 2a, which then outputs masks 3 that designate sets of pixels belonging to different object types, and associated weighting matrices 4 that indicate the discriminative features that characterize the different types of entities in the input image 1.

[0032] In step 120, an encoder network 5 converts the input image 1 and the combined weight matrix 4 into at least one mask representation 6 in a latent space, which is a notion of a class assignment to a mask.

[0033] In step 130, the input image 1 together with the mask representation 6 is processed by the visual processing networks 2b, 2c into refinements 3*, 3** for the mask 3 and refined weight matrices 4*, 4**.

[0034] In step 140, the encoder network 5 converts the refined weight matrix 4*,4** combined with the input image 1 into at least one refined mask representation 6*,6** in latent space.

[0035] In step 150, the sought segmentation 7 is computed from the mask 3 and at least one mask representation 6 in latent space, or from their further refinements 3*, 3**; 6*, 6** obtained by further passing through the vision processing networks 2b, 2c and the encoder network 5.

[0036] According to blocks 111 and 132, at least one of the visual processing networks 2a-2c may be a vision transformer network that computes attention relations between portions of its inputs.

[0037] According to blocks 111a, 132a, weight matrices 4, 4*, 4** can be calculated 111a, 132a from the attention relations.

[0038] According to blocks 121 and 141, an image encoder network that has been trained together with the text encoder network 8 to estimate the best pair between image and text inputs can be selected as the encoder network 5.

[0039] According to blocks 122 and 142, the encoder network 5 can be selected to be a further transformer network that computes attention relationships between portions of its inputs.

[0040] According to blocks 123, 143, the attention bias calculated by the vision transformer network 2a can be applied to at least one attention layer of a further transformer network.

[0041] According to blocks 124 and 144, an image encoder network of a Contrastive Language-Image Pre-training (CLIP) network may be selected as the encoder network 5.

[0042] According to block 131, refinements 3*, 3** for mask 3 may be calculated as offsets to be applied to mask 3, which is calculated first.

[0043] According to block 151, calculating the sought segmentation may include calculating a dot product between at least one mask 3,3*,3** and at least one mask representation 6,6*,6**.

[0044] In addition to simply computing the explored segmentation 7, a class name for the mask 3 can be established. To this end, in the example shown in FIG. 1 , in step 160, a text representation 9a in latent space is computed from the candidate class name 9 using the text encoder network 8 used in training the image encoder network as the encoder network 5. In step 170, this text representation 9a is compared with the mask representation 6,6*,6** in latent space to obtain a comparison result 170a. In step 180, an assignment of the candidate class name 9 to the matching mask 3 is determined from this comparison result 170a. For example, if the representation of the candidate class name 9, "car," in the latent space is very close to the mask representation 6,6*,6**, then it is likely that the mask is related to cars.

[0045] The finally determined segmentation 7 may be used in any suitable manner. In the example shown in Figure 1, in step 190 an operating signal 190a is calculated from the segmentation 7. In step 200, the vehicle 50, the driver assistance system 51, the robot 60, the quality inspection system 70, the monitoring system 80 and / or the medical imaging system 90 are operated by the operating signal 190a.

[0046] FIG. 2 shows how, in one example, an input image can be processed into segments 7 by multiple visual processing network instances 2a-2c.

[0047] All visual processing network instances 2a-2c accept a mask embedding 6,6*,6** as input on top of an input image 1. The first visual processing network 2a processing the input image takes a random initialization I as input instead of the mask embedding 6,6*,6**. The random initialization I generates a mask prediction 3 and an attention bias as a weight matrix 4. This weight matrix 4, together with the input image 1, is processed into a mask embedding 6 by an encoder network 5, here the image encoder of a CLIP network.

[0048] The input image 1 is then fed into the next visual processing network instance 2b, along with the mask embedding 6. This visual processing network instance 2b generates a refinement 3* for the mask (here in the form of an offset to be added to the original mask 3) and a new attention bias as a new weight matrix 4*. This refined weight matrix 4*, along with the input image 1, is again processed by the encoder 5 into a refined mask embedding 6*.

[0049] In the example shown in Figure 2, there is one further iteration with a third visual processing network instance 2c. From the input image 1 combined with the refined mask embedding 6*, this third visual processing network instance 2c generates a second refinement for the mask 3** (again in the form of an additive offset) and yet another attention bias as a new weight matrix 4**. This new weight matrix 4** is combined with the input image 1 and processed by the encoder 5 into the final mask embedding 6**.

[0050] The final segmentation 7 is calculated from a final mask which is the sum of the original mask 3 and the additive offsets 3* and 3** combined with the final mask embedding 6**.

[0051] FIG. 3 illustrates the interaction between the mask 3 and the mask embedding 6 for an example input image 1 showing a crowded street scene. In the input image, some regions belonging to the class “vehicle” are depicted as outlines according to the mask 3, and the corresponding mask embedding 6 is depicted as dots, where the density of the dots corresponds to the value of the mask embedding 6. Because the scene is crowded and contains many other objects, such as pedestrians, near the vehicle, the mask 3 can only provide a rough outline of the vehicle. The mask embedding 6 (e.g., attention bias) derived from the weight matrix 4 enables a finer-grained distinction between the vehicle and other nearby objects. When the dot product of the mask 3 and the mask embedding 6 is calculated, only pixels designated by the mask 3 and that also have high values ​​in the mask embedding 6 are designated as belonging to the class “vehicle.” In particular, because the street scene shown in FIG. 3 is crowded, the mask 3 for the class “vehicle” can easily overlap with masks for nearby object classes, such as “road surface” or “pedestrian.”

[0052] Figure 4 shows how class names can be assigned to masks in an open vocabulary setting. The attention biases as a weight matrix 4, together with the input image 1, are converted by an encoder 5 into a mask representation, which in the example shown in Figure 4, includes a first part 6(3) associated with the first mask 3 and a second part 6'(3') associated with the second mask 3'.

[0053] The encoder network 5 is trained in conjunction with a text encoder network 8 as part of a CLIP network. When a first class name 9, "automobile," is input to the text encoder network 8, the first class name 9 is converted into a first representation 9a in the latent space that is close to the mask representation 6(3) associated with the first mask 3. This provides information that the first mask 3 is most likely associated with the class "automobile."

[0054] Similarly, when a second class name 9', "pedestrian," is input to the text encoder network 8, the second class name 9' is converted into a second representation 9a' in the latent space that is close to the mask representation 6'(3') associated with the second mask 3'. This provides information that the second mask 3' is most likely associated with the class "pedestrian."

Claims

1. A method (100) for determining a segmentation (7) of an input image (1), the input image (1) comprising pixels having pixel values, the segmentation (7) comprising assigning to each pixel of the input image (1) a class of an entity causing the pixel value of that pixel; The method comprises: - providing (110) said input image (1) to a visual processing network (2a), which outputs masks (3) specifying sets of pixels belonging to different object types and associated weight matrices (4), said weight matrices (4) indicating discriminative features characterizing each different type of entity in said input image (1); - transforming (120) said weight matrix (4) combined with said input image (1) into at least one mask representation (6) in a latent space, which is a notion of class assignment to a mask, by an encoder network (5); - processing (130) the input image (1) together with the mask representation (6) by a visual processing network (2b, 2c) into a refinement field (3*, 3**) for the mask (3) and a refined weight matrix (4*, 4**); Transforming (140) the refined weight matrix (4*, 4**) combined with the input image (1) into at least one refined mask representation (6*, 6**) in the latent space by the encoder network (5); a step (150) of calculating the sought segmentation (7) from the mask (3) and the at least one mask representation (6) in the latent space, or from further refinements thereof (3*, 3**; 6*, 6**) obtained by further passing through a vision processing network (2b, 2c) and the encoder network (5); A method comprising:

2. At least one of the visual processing networks (2a-2c) is a vision transformer network (111, 132) that computes attention relations between portions of its input; The method (100) of claim 1.

3. The weight matrix (4, 4*, 4**) is calculated from the attention relations (111a, 132a). The method (100) of claim 2.

4. An image encoder network (5) that has been trained together with a text encoder network (8) to estimate the best pair between image and text inputs is selected (121, 141) as the encoder network (5); The method (100) of any one of claims 1 to 3.

5. The method (100) comprises: Computing (160) text representations (9a) in the latent space from candidate class names (9) using the text encoder network (8); Comparing (170) the text representation (9a) with the mask representation (6,6*,6**) in the latent space; - determining (180) from the results (170a) of the comparison an assignment (180) of the candidate class names (9) to matching masks (3); The method (100) of claim 4, further comprising:

6. The encoder network (5) is selected (122, 142) to be a further transformer network that computes attention relations between parts of its input. The method (100) according to claim 4 or 5.

7. the attention bias calculated by the vision transformer network (2a) is applied to at least one attention layer of the further transformer network (123, 143); The method (100) of claims 3 and 6.

8. The image encoder network of a contrastive language-image pre-training (CLIP) network is selected as the encoder network (5) (124, 144). The method (100) of any one of claims 4 to 7.

9. A refinement (3*, 3**) for said mask (3) is calculated (131) as an offset to be applied to said mask (3) which is initially calculated; The method (100) of any one of claims 1 to 8.

10. Calculating the sought segmentation (7) comprises calculating (151) a dot product between at least one mask (3, 3*, 3**) and at least one mask representation (6, 6*, 6**), The method (100) of any one of claims 1 to 9.

11. a single and identical visual processing network (2a) is used for the initial calculation of the mask (3) and the weight matrix (4) on the one hand, and for their refinement (3*, 3**; 4*, 4**) on the other hand; The functions that the visual processing network (2a) must perform in each application are controlled by additional inputs to the visual processing network (2a); The method (100) of any one of claims 1 to 10.

12. An input image (1) acquired by at least one sensor is selected (105); A motion signal (190a) is calculated (190) from said segmentation (7), a vehicle (50), a driving assistance system (51), a robot (60), a quality inspection system (70), a monitoring system (80), and / or a medical imaging system (90) is operated (200) by said operating signal (190a); The method (100) of any one of claims 1 to 11.

13. 13. A computer program comprising machine-readable instructions that, when executed by one or more computers and / or computing instances, cause the one or more computers and / or computing instances to perform the method (100) of any one of claims 1 to 12.

14. A non-transitory machine-readable storage medium and / or downloadable product comprising the computer program of claim 13.

15. One or more computers and / or computing instances comprising the computer program of claim 13 and / or comprising the non-transitory machine-readable storage medium and / or downloadable product of claim 14.

Images