Method for training a model for semantic segmentation of image data

Abstract

The invention relates to a method for training a model for the semantic segmentation of image data, wherein the method (1) comprises the following steps: - Providing labeled training data for training the semantic segmentation model, wherein the labeled training data comprises image data representing a plurality of environments, and wherein the image data for each of the plurality of environments each comprise a plurality of image data representing the same environment (2); - For each of the plurality of environments, extracting features from the image data that represent the corresponding environment (3); - For each of the multiple environments, fusing the extracted features (4); - Training the model for semantic segmentation based on the labeled training data and the fused extracted features (5); and - Providing the trained machine learning algorithm (6).

Description

[0001] The invention relates to a method for training a model for the semantic segmentation of image data, in which the prediction quality is improved by suitable fusion or bundling of the information contained in different image data.

[0002] Image data refers to data that can be displayed as an image or graphic using a special program.

[0003] Segmentation also refers to the content-based consolidation of connected regions by combining neighboring pixels or voxels according to a specific homogeneity criterion. Semantic segmentation refers, in particular, to the division of image data into different classes.

[0004] Corresponding segmentation models or algorithms for the semantic segmentation of image data are, for example, an important component of image processing in motor vehicles, especially autonomous vehicles. Through pixel-level classification, semantically segmented image data provides a better understanding of the scene than 2D object recognition and serves as a starting point for various driver assistance systems, for example, algorithms in the areas of environment recognition, situation analysis, and trajectory planning.

[0005] However, it turns out to be disadvantageous that image data from different optical sensors, each representing the same scene or the same environment, are semantically segmented separately and independently of each other, whereby some image data, for example due to the arrangement of the corresponding optical sensor, may only contain insufficient information regarding individual classes.

[0006] From the document DE 10 2013 223 803 A1 a method for segmenting an occupancy grid for an environment model of a driver assistance system for a vehicle is known, which method comprises a step of reading in an occupancy grid with a plurality of grid cells, wherein each grid cell of the plurality of grid cells is assigned grid cell information which comprises information about an occupancy level and at least one additional piece of information about the grid cell, as well as a step of assigning at least one object and / or a plurality of objects to the plurality of grid cells using the grid cell information in order to segment the occupancy grid for an environment model.

[0007] The invention is therefore based on the object of providing an improved model for the semantic segmentation of image data.

[0008] The problem is solved by a method for training a model for the semantic segmentation of image data according to the features of patent claim 1.

[0009] The problem is also solved by a system for training a model for the semantic segmentation of image data according to the features of patent claim 8. Disclosure of the invention

[0010] According to one embodiment of the invention, this object is achieved by a method for training a model for the semantic segmentation of image data, wherein labeled training data are provided for training the model for semantic segmentation, wherein the labeled training data comprise image data representing a plurality of environments, and wherein the image data for each of the plurality of environments each comprise a plurality of image data representing the same environment, wherein for each of the plurality of environments, features are extracted from the image data representing the corresponding environment and the corresponding extracted features are respectively fused, wherein the model for semantic segmentation is trained based on the labeled training data and the fused extracted features, and wherein the trained model for semantic segmentation is subsequently provided.

[0011] Training data refers to examples used to learn patterns or relationships in the data. Labeled training data refers to training data further provided with corresponding labels or ground truth information.

[0012] The multiple image data representing the same environment may in particular be images of the same environment from different perspectives and / or with optical sensors arranged at different positions.

[0013] A feature is also understood to be a class or property assigned to a pixel within the image data.

[0014] The fusion of extracted features also means that they are combined or merged. Specifically, a single feature map is created from individual, independent feature maps.

[0015] Thus, a semantic segmentation model is trained that fuses information contained in different images at the feature level, thus achieving improved prediction quality. For example, exchanging semantic information between different perspectives can increase the semantic content of the learned features.

[0016] Overall, a method for training a semantic segmentation model is provided, with which an improved model for semantic segmentation of image data can be trained.

[0017] In one embodiment, the step of fusing the extracted features for each of the plurality of environments comprises minimizing a perspective offset between the corresponding image data.

[0018] A perspective offset is understood to be an offset or distance between the individual image data of the same environment, for example those taken from different perspectives.

[0019] By minimizing the perspective offset, the corresponding data or information can be spatially aligned so that it appears as if it were captured from the same perspective. Subsequently, fused extracted features can be generated by simply summing the individual extracted features and / or convolution operations. Overall, the fused extracted features can thus be generated in a simple manner, without the need for complex and resource-intensive adjustments.

[0020] In a further embodiment, the step of fusing the extracted features for each of the plurality of environments comprises applying a cross attention mechanism.

[0021] A cross attention mechanism is understood to be a mechanism that connects individual image sections or different image data with each other, i.e. transforms the different image data in such a way that joint processing is possible.

[0022] Thus, the fused extracted features can be easily generated using known methods without the need for complex and resource-intensive adaptations.

[0023] The model for semantic segmentation of image data can also be an encoder-decoder structure.

[0024] An encoder-decoder structure is defined as a structure consisting of at least two components: an encoder and a decoder. The encoder converts input data into a representation in a space with lower dimensions, i.e., it compresses the input data into a less complex representation. The decoder, in turn, reconstructs the compressed input data into output data that essentially corresponds to, or most closely resembles, the input data.

[0025] The use of encoder-decoder structures for the semantic segmentation of image data is known, so that this can be applied here in a simple manner, wherein according to the invention a fusion module or a unit which is designed to fuse extracted features is added.

[0026] In addition, the training data may be image data captured by at least two optical sensors of a motor vehicle.

[0027] A sensor, which is also called a detector, (measured variable or measuring) sensor or (measured) probe, is a technical component that can detect certain physical or chemical properties and / or the material properties of its environment qualitatively or quantitatively as a measured variable.

[0028] An optical sensor is understood to be a sensor that is designed to capture image data, for example a camera.

[0029] Models for the semantic segmentation of image data are an important component for image processing in motor vehicles, especially autonomous vehicles.

[0030] A further embodiment of the invention also provides a method for the semantic segmentation of image data, the method comprising capturing a plurality of image data, each representing the same environment, semantically segmenting the captured image data by a model for the semantic segmentation of image data to generate segmentation results, the model for the semantic segmentation of image data having been trained by a method described above for training a model for the semantic segmentation of image data, and providing the segmentation results.

[0031] Thus, a method for semantic segmentation of image data is provided, which is based on an improved model for semantic segmentation of image data trained using a method for training a semantic segmentation model described above. A semantic segmentation model is trained in which information contained in different images is fused at the feature level, thus demonstrating improved prediction quality. For example, the semantic content of the learned features can be increased by exchanging semantic information between different perspectives.

[0032] A further embodiment of the invention also provides a method for controlling a controllable system based on image data, wherein the method comprises capturing a plurality of image data, each representing the same environment with respect to the controllable system, semantically segmenting the captured image data by a method for semantic segmentation of image data as described above, and controlling the controllable system based on the segmented image data.

[0033] A controllable system is understood to be a robotic system, for example an autonomously driving motor vehicle.

[0034] Thus, a method for controlling a controllable system is provided, which is based on an improved model for semantic segmentation of image data, trained by a method for training a semantic segmentation model described above. A semantic segmentation model is trained in which information contained in different images is fused at the feature level, thus exhibiting improved prediction quality. For example, the semantic content of the learned features can be increased by exchanging semantic information between different perspectives.

[0035] A further embodiment of the invention also provides a system for training a model for the semantic segmentation of image data, wherein the system comprises a first provision unit which is designed to provide labeled training data for training the model for semantic segmentation, wherein the labeled training data comprises image data representing a plurality of environments, and wherein the image data for each of the plurality of environments each comprise a plurality of image data representing the same environment, an extraction unit which is designed to extract features from the image data representing the corresponding environment for each of the plurality of environments, a fusion module which is designed to fuse the extracted features for each of the plurality of environments, a training unit which is designedto train the model for semantic segmentation of image data based on the labeled training data and the fused extracted training data, and a second provision unit which is designed to provide the trained model for semantic segmentation of image data.

[0036] Thus, a system for training a semantic segmentation model is provided, which can be used to train an improved model for semantic segmentation of image data. A semantic segmentation model is trained by fusing information contained in different images at the feature level, thus achieving improved prediction quality. For example, the semantic content of the learned features can be increased by exchanging semantic information between different perspectives.

[0037] In one embodiment, the fusion module is designed to minimize perspective offsets between the corresponding image data. Thus, the fused extracted features can be generated easily, without the need for complex and resource-intensive adjustments.

[0038] In a further embodiment, the fusion module is configured to apply a cross-attention mechanism. Thus, the fused extracted features can be easily generated using known methods, without the need for complex and resource-intensive adaptations.

[0039] The model for semantic segmentation of image data can also be an encoder-decoder structure. The use of encoder-decoder structures for semantic segmentation of image data is known, so they can be easily applied here. According to the invention, a fusion module or a unit designed to fuse extracted features is added.

[0040] In addition, the training data can be image data captured by at least two optical sensors of a motor vehicle. Models for the semantic segmentation of image data are an important component for image processing in motor vehicles, especially autonomous vehicles.

[0041] A further embodiment of the invention also provides a system for the semantic segmentation of image data, wherein the system comprises at least one acquisition unit which is designed to acquire a plurality of image data, each representing the same environment, a segmentation unit which is designed to semantically segment the acquired image data by applying a model for the semantic segmentation of image data in order to generate segmentation results, wherein the model for the semantic segmentation of image data is generated by a system described above for training a model for the semantic segmentation of image data, and a provision unit which is designed to provide the segmentation results.

[0042] Thus, a system for the semantic segmentation of image data is specified, which is based on an improved model for the semantic segmentation of image data trained by a system for training a semantic segmentation model described above. A semantic segmentation model is trained in which information contained in different images is fused at the feature level, thus demonstrating improved prediction quality. For example, the semantic content of the learned features can be increased by exchanging semantic information between different perspectives.

[0043] A further embodiment of the invention also provides a system for controlling a controllable system, wherein the system comprises at least one acquisition unit which is designed to acquire a plurality of image data, each representing the same environment with respect to the controllable system, a system for semantic segmentation of image data as described above, which is designed to semantically segment the image data acquired by the at least one acquisition unit in order to generate segmented image data, and a control unit which is designed to control the controllable system based on the segmented image data.

[0044] Thus, a system for controlling a controllable system is provided, which is based on an improved model for semantic segmentation of image data, trained by a system for training a semantic segmentation model described above. A semantic segmentation model is trained in which information contained in different images is fused at the feature level, thus exhibiting improved prediction quality. For example, the semantic content of the learned features can be increased by exchanging semantic information between different perspectives.

[0045] In summary, the present invention provides a method for training a model for semantic segmentation, in which the prediction quality is improved by suitable fusion or bundling of the information contained in different image data.

[0046] The described designs and further training courses can be combined as desired.

[0047] Further possible embodiments, developments and implementations of the invention also include combinations of features of the invention described previously or below with regard to the embodiments that are not explicitly mentioned. Short description of the drawings

[0048] The accompanying drawings are intended to provide a further understanding of embodiments of the invention. They illustrate embodiments and, in conjunction with the description, serve to explain principles and concepts of the invention.

[0049] Other embodiments and many of the aforementioned advantages will become apparent upon review of the drawings. The elements illustrated in the drawings are not necessarily drawn to scale.

[0050] They show: Fig. 1. shows a flowchart of a method for semantic segmentation of image data according to embodiments of the invention; Fig. 2 shows a schematic block diagram of a system for semantic segmentation of image data according to embodiments of the invention; Fig. 3 shows a schematic block diagram of a system for semantic segmentation of image data according to a first embodiment; Fig. 4 shows a schematic block diagram of a system for semantic segmentation of image data according to a second embodiment.

[0051] In the figures of the drawings, the same reference symbols designate the same or functionally equivalent elements, parts or components, unless otherwise stated.

[0052] Fig. 1 shows a flowchart of a method for semantic segmentation of image data 1 according to embodiments of the invention.

[0053] Segmentation models or algorithms for the semantic segmentation of image data, for example, are an important component of image processing in motor vehicles, especially autonomous vehicles. Through pixel-level classification, semantically segmented image data provides a better understanding of the scene than 2D object recognition and serves as a starting point for various driver assistance systems, for example, algorithms in the areas of environment recognition, situation analysis, and trajectory planning.

[0054] However, it turns out to be disadvantageous that image data from different optical sensors, each representing the same scene or the same environment, are semantically segmented separately and independently of each other, whereby some image data, for example due to the arrangement of the corresponding optical sensor, may only contain insufficient information regarding individual classes.

[0055] Fig. 1 shows a method for training a model for semantic segmentation of image data 1, which method comprises a step 2 of providing labeled training data for training the model for semantic segmentation, wherein the labeled training data comprises image data representing a plurality of environments, and wherein the image data for each of the plurality of environments each comprise a plurality of image data representing the same environment, wherein for each of the plurality of environments features are extracted from the image data representing the corresponding environment in a step 3 and the corresponding extracted features are each fused in a step 4, and wherein the model for semantic segmentation is trained in a step 5 based on the labeled training data and the fused extracted features, and wherein the trained model for semantic segmentation is subsequently provided in a step 6.

[0056] Thus, a semantic segmentation model is trained that fuses information contained in different images at the feature level, thus achieving improved prediction quality. For example, exchanging semantic information between different perspectives can increase the semantic content of the learned features.

[0057] Overall, a method for training a semantic segmentation model 1 is provided, with which an improved model for the semantic segmentation of image data can be trained. In particular, a model for the semantic segmentation of image data is trained that enables information exchange, for example, information exchange between images of the same environment taken from different perspectives, i.e., parallel processing of multiple images of the same environment in a single step. This is achieved by learning how to exchange semantics between different data, for example, different perspectives, during training.In particular, the training algorithm for training a model for semantic segmentation of image data is adapted in such a way that each training step contains several images of the same environment, for example the images of all perspectives to be fused.

[0058] In particular, fusion can be performed on any learned fusion or any possible fusion module having learnable parameters, whereby the parameters can be adjusted accordingly during a training phase.

[0059] According to the embodiments of the Fig. 1, step 4 of fusing the extracted features involves minimizing a perspective offset between the corresponding image data.

[0060] According to the embodiments of the Fig. 1, step 4 of fusing the extracted features also involves applying a cross attention mechanism.

[0061] In particular, step 4 of merging the extracted features can be designed in such a way that new feature images are generated which have a higher semantic content.

[0062] According to the embodiments of the Fig. 1, the model for semantic segmentation of image data is an encoder-decoder structure.

[0063] In addition, the training data consists of image data captured by at least two optical sensors of a motor vehicle.

[0064] In particular, a model for the semantic segmentation of image data can be trained, which model is designed to fuse environmental images of a motor vehicle, for example an autonomously driving motor vehicle, which has several (front) cameras that are mounted at different vehicle positions and thus record different partial sections of the same scene or environment.

[0065] Fig. 2 shows a schematic block diagram of a system for semantic segmentation of image data 10 according to embodiments of the invention.

[0066] How Fig. 2 shows, the system 10 has acquisition units which are designed to acquire a plurality of image data 11 each representing the same environment, a model for the semantic segmentation of image data or a segmentation unit 12 which is designed to semantically segment the acquired image data 11 by applying a model for the semantic segmentation of image data in order to generate segmentation results 13 or segmentation masks, and a provision unit which is designed to provide the segmentation results 13.

[0067] The segmentation unit can be implemented, for example, based on code stored in a memory and executable by a processor. The provision unit can also be a transmitter configured to transmit corresponding data.

[0068] According to the embodiments of the Fig. 2, the segmentation unit was trained based on labeled training data for training the model for semantic segmentation, wherein the training data comprises image data representing a plurality of environments, and wherein the image data for each of the plurality of environments each comprise a plurality of image data representing the same environment, in such a way that an extraction unit 14 is designed to extract features from the image data representing the corresponding environment for each of the plurality of environments, a fusion module 15 is designed to fuse the extracted features for each of the plurality of environments, and a reconstruction unit 16 is designed to generate segmentation results based on the fused extracted features.

[0069] The segmentation unit has an encoder-decoder structure. The extraction unit 14 forms the encoder, and the reconstruction unit forms the decoder. According to the invention, the fusion module 15, i.e., a unit configured to fuse image features calculated based on different camera perspectives, is added.

[0070] The fusion module can also be implemented based on code stored in a memory and executable by a processor.

[0071] According to the embodiments of the Fig. 2, the input data is again image data captured by at least two optical sensors of a motor vehicle.

[0072] Fig. 3 shows a schematic block diagram of a system for semantic segmentation of image data 20 according to a first embodiment.

[0073] How Fig. 3 shows, the system 20 in turn has an extraction unit 21 which is designed to extract features from the image data representing the corresponding environment for each of a plurality of environments in image data, wherein the image data has image data representing a plurality of environments, and wherein the image data for each of the plurality of environments has a plurality of image data representing the same environment, a fusion module 22 which is designed to fuse the extracted features for each of the plurality of environments, and a reconstruction unit 23 which is designed to generate segmentation results based on the fused extracted features.

[0074] According to the first embodiment, the fusion module 22 comprises an offset unit 24, which is designed to minimize a perspective offset between the corresponding image data, a scaling unit 25, which is designed to additionally scale the corresponding pixels with a learnable parameter such that pixels from different cameras lie on the same value range, and a generation unit 26, which is designed to generate a common feature map by adding the corresponding transformed and scaled independent feature maps or by applying convolution operations to the corresponding transformed and scaled independent feature maps.

[0075] According to the first embodiment, the features extracted from image data of a front camera or the corresponding feature map are also taken into account again in the generation unit 26 and are included in the generation of the common feature map.

[0076] The offset unit, the scaling unit and the generation unit can each be implemented, for example, based on code stored in a memory and executable by a processor.

[0077] Fig. 4 shows a schematic block diagram of a system for semantic segmentation of image data 30 according to a first embodiment.

[0078] How Fig. 4 shows, the system 30 in turn has an extraction unit 31 which is designed to extract features from the image data representing the corresponding environment for each of a plurality of environments in image data, wherein the image data has image data representing a plurality of environments, and wherein the image data for each of the plurality of environments has a plurality of image data representing the same environment, a fusion module 32 which is designed to fuse the extracted features for each of the plurality of environments, and a reconstruction unit 33 which is designed to generate segmentation results based on the fused extracted features.

[0079] According to the second embodiment, the fusion module 32 is designed to apply a cross-attention mechanism. Fig.4 shows, the fusion module 32 has, in particular, an application unit 34, which is designed to apply a multi-head attention mechanism to the individual, independent feature maps, i.e., to find matching images, wherein the individual, independent feature maps are numbered or made distinguishable by a positioning unit 35. The outputs of the application unit 34 are additionally processed by a processing unit 36, wherein the processing unit 36 ​​is designed, for example, to correct the data by applying a feedforward neural network, and a generation unit 37, which is designed to generate a common feature map by adding together the corresponding outputs of the processing unit 36.According to the second embodiment, the features extracted from image data of a front camera or the corresponding feature map are also taken into account again in the generation unit 37 and are included in the generation of the common feature map.

[0080] The application unit, the positioning unit, the processing unit and the generation unit can each be implemented, for example, based on a code stored in a memory and executable by a processor. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] DE 10 2013 223 803 A1

[0006]

Claims

[1] A method for training a model for semantic segmentation of image data, the method (1) comprising the following steps: - Providing labeled training data for training the semantic segmentation model, wherein the labeled training data comprises image data representing a plurality of environments, and wherein the image data for each of the plurality of environments each comprise a plurality of image data representing the same environment (2); - For each of the plurality of environments, extracting features from the image data that represent the corresponding environment (3); - For each of the multiple environments, fusing the extracted features (4); - Training the model for semantic segmentation based on the labeled training data and the fused extracted features (5); and - Providing the trained machine learning algorithm (6). [2] The method (1) of claim 1, wherein the step of fusing the extracted features (4) for each of the plurality of environments comprises minimizing a perspective offset between the corresponding image data. [3] Method (1) according to claim 1 or 2, wherein the step of fusing the extracted features (4) for each of the plurality of environments comprises applying a cross attention mechanism. [4] Method (1) according to one of claims 1 to 3, wherein the model for the semantic segmentation of image data is an encoder-decoder structure. [5] Method (1) according to one of claims 1 to 4, wherein the training data are image data acquired by at least two optical sensors of a motor vehicle. [6] Method for the semantic segmentation of image data, the method comprising the following steps: - Capturing multiple image data, each representing the same environment; - Semantically segmenting the acquired image data by a model for semantic segmentation of image data to generate segmentation results, wherein the model for semantic segmentation of image data has been trained by a method for training a model for semantic segmentation of image data according to one of claims 1 to 5; and - Providing the segmentation results. [7] A method for controlling a controllable system based on image data, the method comprising the following steps: - Acquiring multiple image data, each representing the same environment with respect to the controllable system; - segmenting the acquired image data by a method for semantic segmentation of image data according to claim 6; and - Controlling the controllable system based on the segmented image data. [8] System for training a model for semantic segmentation of image data, wherein the system comprises a first provision unit which is designed to provide labeled training data for training the model for semantic segmentation, wherein the labeled training data comprises image data representing a plurality of environments, and wherein the image data for each of the plurality of environments each comprise a plurality of image data representing the same environment, an extraction unit which is designed to extract features from the image data representing the corresponding environment for each of the plurality of environments, a fusion module which is designed to fuse the extracted features for each of the plurality of environments, a training unit which is designed,to train the model for semantic segmentation of image data based on the labeled training data and the fused extracted training data, and a second provision unit which is designed to provide the trained model for semantic segmentation of image data. [9] The system of claim 8, wherein the fusion module is configured to minimize a perspective offset between the corresponding image data. [10] System according to claim 8 or 9, wherein the fusion module is configured to apply a cross attention mechanism. [11] System according to one of claims 8 to 10, wherein the model for semantic segmentation of image data is an encoder-decoder structure. [12] System according to one of claims 8 to 11, wherein the training data is image data acquired by at least two optical sensors of a motor vehicle. [13] System for the semantic segmentation of image data, wherein the system (10, 20, 30) comprises at least one acquisition unit which is designed to acquire a plurality of image data each representing the same environment, a segmentation unit (12) which is designed to semantically segment the acquired image data by applying a model for the semantic segmentation of image data in order to generate segmentation results, wherein the model for the semantic segmentation of image data was trained by a system for training a model for the semantic segmentation of image data according to one of claims 8 to 12, and a provision unit which is designed to provide the segmentation results. [14] System for controlling a controllable system, wherein the system comprises at least one acquisition unit which is designed to acquire a plurality of image data each representing the same environment with respect to the controllable system, a system for semantic segmentation of image data according to claim 13, which is designed to semantically segment the image data acquired by the at least one acquisition unit in order to generate segmented image data, and a control unit which is designed to control the controllable system based on the segmented image data.

Images