Method for analyzing an environment by means of a capturing device, computer program product, computer-readable storage medium, and capturing device
By analyzing multi-level feature maps with AI to generate and merge kernels with trust values, the method addresses the lack of confidence scoring in current algorithms, ensuring reliable and safe object segmentation for safety-critical applications.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-09-11
- Publication Date
- 2026-04-02
AI Technical Summary
Current image segmentation algorithms lack explainability and confidence scoring, making them unsuitable for safety-critical applications, as they do not account for the confidence level of partial results, leading to potential misjudgments and safety-critical events.
A method using a multi-level and multi-scale feature map analysis with artificial intelligence to generate kernels for object centers, determine confidence levels, and merge these kernels into a fusion kernel with trust values, providing a traceable confidence score for improved object segmentation.
The method enables traceable confidence scoring, allowing safe and reliable object identification for safety-critical applications by ensuring that only confidently detected objects are considered, reducing the likelihood of misjudgments.
Smart Images
Figure EP2025075873_02042026_PF_FP_ABST
Abstract
Description
[0001] 202412630
[0002] 1
[0003] Description
[0004] Method for analyzing an environment using a detection device, computer program product, computer-readable storage medium and detection device
[0005] The following invention relates to a method for analyzing an environment by means of a detection device according to claim 1. The invention further relates to a corresponding computer program product, a corresponding computer-readable storage medium and a detection device.
[0006] In particular, so-called instance segmentation is used to uniquely classify objects and areas of a background, for example grass or sky, in an image in order to obtain an improved image in which each object and each background area is marked by a mask that indicates which pixels of the image belong to an individual object.
[0007] Current algorithms determine exactly one mask for each object to identify its location in the image, as well as a single confidence score for the detected instance of the object. However, the confidence score of the result is often unexplainable and not directly related to any actual physical or real-world quantity, but is usually just a metric based on the losses of the final prediction. In such cases, it does not take into account the confidence level of partial results, which would be important for determining the outcome. Therefore, the confidence score is unusable for safety-related applications, for providing safety evidence, or for making safety-critical decisions.
[0008] Even in real-world situations, there is a degree of ambiguity, particularly at object boundaries or when objects overlap, for example, due to perspective. Depending on the application, incorrect segmentation of image parts can lead to misjudgments of the situation. For instance, if an object recognition system assumes the object is smaller, perhaps with fewer consequences in the event of a collision, or appears outside the collision zone when a collision actually occurs, the object recognition system can provide information that leads to safety-critical events.
[0009] The current state of the art focuses on improving the performance of segmentation systems. Since solutions based on artificial intelligence are described in 202412630
[0010] 2
[0011] Since these factors are generally inexplicable, they are used in applications without security relevance. In these cases, good performance is sufficient without proof that the segmentation system is correct. A precise specification of the confidence level is not required, as the application simply uses the output of the segmentation system without evaluating the confidence level for individual objects.
[0012] Therefore, there is a need, given the current state of the art, to use segmentation systems based on artificial intelligence in safety-relevant applications as well.
[0013] The object of the present invention is to provide a method, a computer program product, a computer-readable storage medium and a recording device by means of which an improved and safe analysis of an environment can be realized.
[0014] This problem is solved by a method, a computer program product, a computer-readable storage medium, and a recording device according to the independent claims. Advantageous embodiments are specified in the dependent claims.
[0015] One aspect of the invention relates to a method for analyzing an environment using a detection device. An image of the environment is captured using an optical detection element of the detection device. A multi-level and multi-scale feature map of the image is generated using artificial intelligence within an electronic computing unit of the detection device. Each level of the multi-level and multi-scale feature map is analyzed by the artificial intelligence with respect to an object center and / or background within that level. A kernel is generated for each detected object within the feature map at each level. A confidence level regarding the object center is determined during the analysis of each level by the artificial intelligence.Each kernel is linked to its corresponding trust value using artificial intelligence. The respective kernels of each stage, along with their respective trust values, are then merged into a single fusion kernel, also with a single trust value, using artificial intelligence. This fusion kernel, along with its associated trust values, is then output for analysis of the environment using artificial intelligence.
[0016] In particular, the proposed solution has the advantage that a trust value is used for a single object and all available information on the way to 202412630 is taken into account.
[0017] 3
[0018] The object identification can be used in a traceable manner. In this way, the confidence score is usable for both safety decisions and safety verifications. At early stages, no objects that are likely to be relevant, and especially those that can be used to determine safety thresholds, are discarded. Hypotheses about objects can be derived based on object safety criteria. For example, different thresholds for the agreement of objects at multiple levels can be formulated. This information is also provided with confidence scores and can be used by subsequent applications with additional information, for example, from various sensors, to determine the best agreement.For each object, the confidence in the final prediction is specified along with the confidence of the underlying kernel, either separately or combined, so that it can be used for further potentially security-relevant decisions.
[0019] In other words, each stage can be passed through a so-called position header element to detect object centers in the background. These can be considered the best estimates and are used to generate suitable kernels with the artificial intelligence's kernel generator. To improve overall performance, the kernels of object candidates that are close enough to each other, particularly using cosine similarity, can be merged using an average clustering algorithm. A final prediction then uses the merged kernels, or fusion kernel, on the result feature encoder component, specifically a high-resolution coded feature map, to obtain pixel-by-pixel indications of whether they belong to a particular unique object.In a final post-processing step, this information is summarized, for example by selecting the object with the maximum value for a given pixel, so that a single segmentation map is returned, with each pixel of this segmentation map assigned to an object.
[0020] In particular, an adaptation of a neural network, especially a so-called Panoptic FON, can be proposed to ensure trustworthiness in secure applications by providing object-level trust values whose generation is understandable and traceable. 202412630
[0021] 4
[0022] Furthermore, the possibility is offered to generate multiple hypotheses for objects that are parameterizable in order to generate more hypotheses, which may lead to a less likely occurrence of an object being neglected.
[0023] The derivation of confidence values for the merged kernels, i.e., the fusion kernel, is based primarily on the prediction of object centers using the position header element, along with a loss-based confidence value for each predicted object center. This confidence value is retained and passed on to the kernel fusion along with the object center. The goal is to obtain a verifiable confidence value for the merged kernel. For kernel fusion, all relevant kernels to be merged into the final kernel for a single object are determined using, for example, cosine similarity. The weights used for the average clustering method for merging the kernels are then used to obtain the confidence values of each kernel.
[0024] For uncorrelated features, the individual weights can be added quadratically, that is, the square root of the sum of the squares is used. For highly correlated features, the weights can be added linearly. In the subsequent steps, all merged kernels whose confidence level exceeds a chosen threshold, which is aligned with security considerations so that no relevant objects are prematurely discarded, are used to determine the final segmentation prediction for the underlying object candidate.
[0025] In object recognition, a kernel refers to a small region within a larger image or scene that serves as the basis for feature or object detection. A kernel can be thought of as a filter or mask that slides over the input image, capturing local information. In image processing and computer vision, kernels are frequently used in the form of convolutional filters to detect features such as edges, textures, and contours. One example of a kernel is the Sobel operator, used to detect horizontal or vertical edges. In machine learning (ML) models for object recognition, such as in convolutional neural networks (CNNs), kernels play a crucial role in feature extraction and processing. In these neural networks, convolutional kernels are used to identify and encode local features in input data.These kernels consist of weights that are optimized during the training process to improve the model's performance. Overall, kernels thus contribute to reducing the complexity of the 202412630.
[0026] 5
[0027] To reduce the object recognition task by capturing and processing local features in input data, resulting in more efficient and accurate object recognition.
[0028] In a particularly advantageous embodiment, the multi-level and multi-scale feature map is further analyzed using a feature encoder. A feature encoder is a component of a machine learning model, especially in artificial intelligence, that compresses and represents information from input data to make it accessible for subsequent processing steps. The encoder converts the original data, for example, pixel values in an image, into a lower dimension by extracting features and eliminating redundant information. This process is also known as feature extraction or reduction. Feature encoders are frequently used, particularly in object recognition algorithms and image processing, to encode pixel or voxel data into a more compact format suitable for tasks such as object detection, segmentation, and classification.By using a feature encoder, the amount of data is reduced, and the storage and computational effort for subsequent processing steps is decreased. The encoder helps to make the machine learning model more efficient and accurate by verifying and encoding the relevant features in the input data. In particular, the feature encoder performs object classification. Specifically, the object classification performed by the feature encoder can then be combined with the object classification resulting from the analysis based on the fusion kernel.
[0029] It has also proven advantageous to determine a pixel's object membership at each stage depending on the specific fusion kernel and the feature encoder analysis. In particular, object membership can thus be determined based on both the fusion kernel and the feature encoder analysis. This allows for the provision of a robust object recognition algorithm.
[0030] In a further advantageous embodiment, the specific object memberships of each level are condensed for the feature map, and a single segmentation map is generated based on this condensation. In particular, the output of the machine learning model or artificial intelligence can thus be the single segmentation map. This single segmentation map then displays the condensation of the individual levels. Specifically, the single segmentation map can therefore show the corresponding object membership. Advantageously, 202412630
[0031] Section 6 now provides that, in addition to the individual segmentation map, a corresponding trust value for a recognized object within the individual segmentation map is also output. Thus, the individual segmentation map can be passed on to subsequent processing units, so that not only a single trust value is passed on with the segmentation map, but, in particular, at least one corresponding trust value for each specific object is passed on, which can then be used additionally in further processing.
[0032] It has also proven advantageous to determine a single confidence level for each segmentation map, at least in relation to the confidence levels of the respective fusion kernels. In particular, a single confidence level for the segmentation map can thus be generated based on the multitude of confidence levels determined, especially at the various stages. This allows the individual segmentation map to be assigned a confidence level, enabling subsequent processing units, for example, to recognize the degree of confidence in the result of that specific segmentation map.
[0033] In a further advantageous embodiment, the individual segmentation map, along with at least the respective confidence values, is transmitted to a higher-level electronic computing unit for further processing. For example, the higher-level electronic computing unit can be combined with other sensors. For instance, the information from a LiDAR sensor can then be used and fused with the information from the individual segmentation map via the optical detection device. If, for example, the LiDAR device arrives at a similar result to the optical detection device or the result of the individual segmentation map, object classification can be improved accordingly.For example, if the LiDAR sensor produces a different result than determined by the individual segmentation map, the confidence level can be used, so that, for instance, the LiDAR sensor is given more trust than the optical detection device. The higher-level electronic computing unit can, for example, be part of a vehicle's driver assistance system.
[0034] It is also advantageous if, depending on a risk-based criterion, a number of kernels for each level and a number of associated confidence values are generated.
[0035] 7. This criterion can assess from a security perspective whether a kernel can be rejected. Only kernels that appear permissible from a security standpoint according to the above criterion are rejected. Alternatively or additionally, a maximum number of kernels can be specified. For example, only 50 kernels can be specified for each level. Kernels with a correspondingly high confidence level can then be used accordingly. Kernels with a low confidence level, and therefore not among the 50 kernels, can be rejected. A threshold for the confidence level can be defined application-specifically. This allows the analysis to be performed computationally efficiently.
[0036] In a further advantageous embodiment, kernels that are sufficiently close to each other are merged for each stage. In particular, they can be adjacent, for example. Neighboring kernels can thus be combined. This enables computationally efficient analysis of the environment.
[0037] It has proven advantageous to use cosine similarity of the kernels for fusion. Cosine similarity, also known as cosine similarity, is a measure used to quantify the similarity between two vectors. It calculates the cosine of the angle between two non-zero vectors in multidimensional space and returns a numerical value in the range of [-1, 1]. Cosine similarity is often used in image processing and machine learning to measure the similarity between objects or data points. The smaller the angle between two vectors, the greater their cosine similarity and the more similar they are to each other. The formula for calculating cosine similarity is: cos(0) = (A * B) / (||A|| * ||B||)
[0038] Here, A and B are two vectors, the multiplication sign (*) denotes the dot product, and the double bar is the vector's length norm. Cosine similarity has several advantages over other similarity measures, such as Euclidean distance. Euclidean distance is robust to different dimensional scales and considers only the direction of the vectors, not their length. This prevents vectors from being classified as dissimilar simply because they are on different scales. 202412630
[0039] 8
[0040] It has proven advantageous to use a feature pyramid network to generate the multi-level, multi-scale feature map. The feature pyramid network, also known as an FPN, is a deep neural network used for object detection and semantic segmentation in computer vision. It is designed to overcome the challenge of varying the size and shape of objects in input images, which can lead to poor performance in detecting small or obscured objects. The FPN is a multi-level network that creates a "pyramid" of feature maps at different scales, generated at various levels of the base network. This pyramid enables the network to detect objects of different sizes by combining the features of the same objects at different scales.The FPN also uses top-down connections from higher to lower levels to transfer information from the upper levels, which have high resolution and semantic strength, to the lower levels, which have lower resolution but finer features. The use of FPN has proven effective in improving object detection and semantic segmentation performance in computer vision, especially in the detection of small or hidden objects.
[0041] It has proven advantageous to deploy artificial intelligence as a fully convolutional network for panoptic segmentation. Specifically, this network is also known as the Fully Convolutional Network for Panoptic Segmentation (FCN for panoptic segmentation). This is a deep neural network used to perform instance and semantic segmentation simultaneously. Panoptic segmentation is a task in computer-aided visual perception where each pixel in an input image is assigned to a specific class, and instances of each class are simultaneously identified. To achieve this, panoptic segmentation combines the results of semantic segmentation, particularly pixel classification, and instance segmentation, specifically the identification and segmentation of individual instances of the same class.The FCN for panoptic segmentation consists of two main components. The first is the backbone network. This can be a pre-trained deep convolutional network, such as RESNET or VGG, which is used to maintain functionality. The backbone network takes an input image and generates a series of feature maps with decreasing spatial resolution and increasing semantic significance. In this case, the feature pyramid network is used. The second part is the segmentation decoder. The segmentation decoder is a specially designed 202412630.
[0042] 9
[0043] This module builds upon the backbone network and translates the feature maps into class masks for each pixel of the input image. The decoder can consist of several convolution-based blocks used to improve spatial resolution, as well as a final pixel classifier that generates class labels for each pixel. To perform panoptic segmentation, the FCN is divided into two stages. First, semantic segmentation is performed, in which the FCN initially creates semantic segmentation for the pixel in the input image. For this, the feature maps of the backbone network are used to predict class labels for the pixel. Instance segmentation follows semantic segmentation and is used to identify and segment individual objects of the same class within the instances.Semantic segmentation is combined with a clustering algorithm to separate instances and generate a final panoptic segmentation. In summary, a complete convoluted panoptic segmentation network simultaneously performs semantic and instance segmentation by combining deep learning and feature engineering to classify pixels in input images and separate objects.
[0044] In a further advantageous embodiment, it is provided that at least one hypothesis for each kernel is additionally determined. In particular, the respective hypotheses (per pixel) for the different potentially detected objects can be output, so that subsequent applications can use them and no information is lost, for example, if a pixel / area was only narrowly assigned to one object but could be assigned to another with a similar probability. The hypotheses for objects can be generated in such a way that they are parameterizable to generate more hypotheses, which can lead to a reduction in the likelihood of an object being overlooked.
[0045] For example, the detection device can be designed to analyze the surroundings of a motor vehicle, particularly a train, or a work environment. A motor vehicle could also be, for example, a two-wheeler or a car. Furthermore, the detection device can also be used to monitor a workplace. Thus, the detection device is highly versatile. The detection unit itself could, for example, be a camera.
[0046] The presented method is, in particular, a computer-implemented method. Therefore, another aspect of the invention relates to a computer program product with 202412630
[0047] 10
[0048] Program code means which cause an electronic computing device, when the program code means are processed by the electronic computing device, to carry out a procedure according to the preceding aspect.
[0049] A further aspect of the invention relates to a computer-readable storage medium containing at least one computer program product according to the preceding aspect.
[0050] Furthermore, the invention relates to a detection device for analyzing an environment, comprising at least one optical detection device and an electronic computing device, wherein the detection device is configured to carry out a method according to the preceding aspect. In particular, the method is carried out by means of the detection device.
[0051] A computing unit / electronic computing device can be understood, in particular, as a data processing device containing a processing circuit. The computing unit can therefore process data to perform arithmetic operations. This may also include operations to perform indexed access to a data structure, such as a lookup table (LUT).
[0052] The computing unit may, in particular, contain one or more computers, one or more microcontrollers, and / or one or more integrated circuits, for example, one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), and / or one or more systems on a chip (SoCs). The computing unit may also contain one or more processors, for example, one or more microprocessors, one or more central processing units (CPUs), one or more graphics processing units (GPUs), and / or one or more signal processors, in particular one or more digital signal processors (DSPs). The computing unit may also include a physical or virtual array of computers or other units of the aforementioned type.
[0053] In various embodiments, the computing unit includes one or more hardware and / or software interfaces and / or one or more memory units. 202412630
[0054] 11
[0055] A storage unit can be volatile data storage, for example as dynamic random access memory (DRAM) or static random access memory (SRAM), or as non-volatile data storage, for example as read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or flash EEPROM, ferroelectric random access memory (FRAM), or magnetoresistive random access memory.It can be designed as MRAM (magnetoresistive random access memory) or as phase-change random access memory, PCRAM (phase-change random access memory).
[0056] Here and in the following, an artificial neural network can be understood as software code stored on a computer-readable storage medium that represents one or more interconnected artificial neurons or can replicate their function. The software code can also contain multiple software code components, which may, for example, have different functions. In particular, an artificial neural network can implement a nonlinear model or a nonlinear algorithm that maps an input to an output, where the input is given by an input feature vector or an input sequence, and the output may, for example, include a category for a classification task, one or more predicted values, or a predicted sequence.
[0057] Computer vision deals specifically with the automatic processing of visual information. The goal of computer vision is to teach computer systems to recognize, classify, and react to objects, people, scenes, and actions in digital images and videos. Computer vision systems use algorithms and techniques from the fields of machine learning, pattern recognition, and signal processing to process and interpret visual information. This includes, in particular, image preprocessing, which involves improving and normalizing images, for example, noise reduction, color balance, angle correction, and scaling. Another aspect is feature extraction, which encompasses the recognition and extraction of relevant features from images, such as edges, textures, shapes, and colors. Furthermore, 202412630
[0058] 12. Object recognition is performed, which includes in particular the identification and classification of objects in the images, for example, faces, vehicles, buildings, and landscapes. Scene analysis is also part of the process, encompassing the analysis and interpretation of scenes, such as the recognition of actions, movements, and relationships between objects. The use of machine learning algorithms is specifically intended to train computer vision systems and improve their performance. Computer vision, and thus also the method according to the invention, has numerous applications in various fields such as robotics, medicine, agriculture, security, entertainment, transportation, and many others.Through the further development of techniques and algorithms, computer vision is becoming increasingly powerful and can automate and simplify complex tasks, which is why such techniques can be reliably used, especially in the field of motor vehicles, and particularly in the field of trains.
[0059] For use cases or application situations that may arise in a method according to the invention and that are not explicitly described herein, it may be provided that, according to the method, an error message and / or a request for user feedback is issued and / or a default setting and / or a predetermined initial state is set.
[0060] Regardless of the grammatical gender of a particular term, persons with male, female or other gender identities are included.
[0061] Further features and combinations of features of the invention will become apparent from the figures and their descriptions, as well as from the claims. In particular, further embodiments of the invention need not necessarily include all features of any one of the claims. Further embodiments of the invention may have features or combinations of features that are not mentioned in the claims.
[0062] The single figure shows a schematic block diagram according to an embodiment of a motor vehicle with an embodiment of a detection device.
[0063] In the figure, identical or functionally equivalent elements are labelled with the same reference symbols. 202412630
[0064] 13
[0065] FIG 1 shows a schematic block diagram according to an embodiment of a motor vehicle 10 with an embodiment of a detection device 12. In the present embodiment, the detection device 12 has, in particular, at least one detection element 14, especially an optical detection element 14, for example in the form of a camera. Furthermore, an electronic computing device 16 is shown.
[0066] The optical detection device 14 is specifically designed to detect an environment 18. In particular, this environment 18 is that of the motor vehicle 10.
[0067] FIG. 1 shows in particular that an image 20 can be provided by the acquisition device 14. In the present embodiment, the image 20 is transferred to a feature pyramid network 22. A multi-level and multi-scale feature map 24 is generated from the feature pyramid network 22, whereby a single level 26 is analyzed accordingly. For this purpose, a so-called kernel generator 28 can be provided. The kernel generator 28 is provided for generating a plurality of kernels 30. The kernels 30 can in turn be fused to form a fusion kernel 32. Furthermore, FIG. 1 shows a high-resolution feature map 34 and a feature encoder 36. In addition, a prediction module 38 and a single segmentation map 40 are shown.
[0068] The kernel generator 28 in turn includes the analysis of object centers 42 and a background 44. Furthermore, a respective confidence value 46 can be determined for each kernel 30 in the kernel generator 28.
[0069] In particular, FIG. 1 shows the detection device 12, which is configured to analyze the environment 18. The image 20 of the environment 18 is captured by the optical detection device 14. The multi-level and multi-scale feature map 24 of the image 20 is generated by an artificial intelligence of the electronic computing device 16. Each level 26 of the multi-level and multi-scale feature map 24 is analyzed with respect to the object center 42 in the respective level 26 and / or the background 44 in the respective level 26. A respective kernel 30 is generated for each detected object center 42 in each level 26 for the feature map 24. A respective confidence value 46 with respect to the object center 42 is determined by the artificial intelligence during the analysis of each level 26. The respective kernel 30 is accompanied by the corresponding 202412630
[0070] 14
[0071] A trust value of 46 is linked using artificial intelligence. The respective kernels 30 of a level 26, each with a trust value of 46, are merged by artificial intelligence to form a fusion kernel 32 with a trust value of 54. This fusion kernel 32, with its associated trust values of 46, is then output to analyze the environment 18 using artificial intelligence. The trust values of 46 are determined in a traceable manner, for example, according to the weights used in the fusion of the kernels 30.
[0072] It is specifically provided that the multi-level and multi-scale feature map 24 is additionally analyzed using the feature encoder 36. This allows the object membership of a pixel in the image 20 to be determined at each level 26, depending on the respective fusion kernel 32 and the analysis by the feature encoder 36. Furthermore, the determined object memberships of each level 26 for the feature map 24 can be condensed, and the individual segmentation map 40 can be generated accordingly. For each segmentation map 40, a single confidence value 48 can be determined based on the respective confidence values 46 of the respective fusion kernels 32. The individual segmentation map 48, along with at least the respective confidence values 46, can be transmitted to a higher-level electronic computing unit 50 for further processing.
[0073] Furthermore, it can be provided that for each level 26, a maximum number of kernels 30 are generated with corresponding confidence values of 46. For each level 26, kernels 30 that are sufficiently close to each other can be merged. This fusion can be achieved, in particular, by means of a cosine similarity of the kernels 30.
[0074] In particular, FIG 1 shows that each individual stage feature is passed through a position header element 52 to detect the object centers 42 and background regions 44. These can be considered best estimates and are used to generate suitable kernels 30 with the kernel generator 28. To improve overall performance, the kernels 30 of object candidates that are sufficiently close to each other, especially using cosine similarity, are merged by an average clustering algorithm. For example, the top n=50 kernels 30 can be retained via a final instance prediction. All other kernels 30 are discarded. The final prediction uses the merged kernels 30, i.e., the fusion kernel 32, on the result feature encoder component, a high-resolution coded 202412630
[0075] 15
[0076] A feature map is created to obtain pixel-by-pixel information about whether a pixel belongs to a specific, unique object. In a final post-processing step, this information is aggregated, for example, by selecting the object with the maximum value for a given pixel, so that a single segmentation map is returned, where each pixel is assigned to an object.
[0077] According to the invention, the panoptic FON is adapted to ensure trustworthiness in secure applications by providing object-level trust values 46, the generation of which is understandable and traceable. Furthermore, the invention offers the possibility of generating multiple hypotheses for objects, which are parameterizable to generate more hypotheses, thereby reducing the likelihood of an object being overlooked.
[0078] The prediction of the object centers 42 by the position header element 52 yields predictions along with a loss-based confidence value 46 for each predicted object center 42. This confidence value 46 is retained and passed on with the object center 42 in the fusion of the kernels 30. The goal is to obtain a traceable confidence value 46 for the fused kernel, i.e., the fusion kernel 32. For the kernel fusion, all relevant kernels 30 to be merged into the final kernel, i.e., the fusion kernel 32, for a single object are identified using cosine similarity. The weights available for the average clustering method for merging the kernels 30 are used to obtain the confidence values 46 of each kernel 30.
[0079] For uncontrolled features, the individual weights can be added quadratically; that is, in particular, the square root of the sum of the squares can be used. For highly correlated features, the weights can be added linearly. In the following steps, the fusion kernels 32, whose confidence level 46 is above a chosen threshold that is aligned with safety aspects so that no relevant objects are prematurely discarded, are used to determine the final prediction of the segmentation for the underlying object candidate.
[0080] Alternatively, the fusion can be derived based on several thresholds for the similarity matrix of the object candidates. Furthermore, there are various combinatorial possibilities to utilize the information present in the individual stage features. There could be a limitation, only 202412630
[0081] 16
[0082] To merge object candidates from "adjacent n levels," for example n = 2, 3, or all possible subsets of combinations of levels. In this way, different hypotheses will potentially point to the same object.
[0083] A confidence score can then be determined for the final prediction step. In the final prediction step, each Fusion kernel 32 is used on the coded function, and a confidence score for the prediction is obtained. This score can then be adjusted against the confidence score 46 of kernel 30, for example, by multiplication, to report a corresponding confidence score for the object that includes the intermediate steps as well as the final prediction. Alternatively, the prediction confidence score can be returned as a tuple that specifies both the value and the confidence score 46 of kernel 30, so that later components can use this information to their advantage. For example, objects that have the same prediction confidence score but have more confidence in kernel 30 will be trusted more.
[0084] As a final step, the instance segmentation map is returned, containing pixel-level information for all kernels. Instead of simply providing information about belonging to the object with the highest score—specifically, a state-of-the-art segmentation map—a layered single segmentation map of all relevant objects is returned, up to a chosen confidence threshold, or the threshold for considering the probability of discarding critical safety-related objects. This allows downstream components to use this information to determine the most likely hypothesis, taking into account additional information such as LiDAR data.
[0085] 202412630
[0086] 17
[0087] Reference symbol list
[0088] 10 motor vehicle
[0089] 12 Detection device
[0090] 14 Recording device
[0091] 16 electronic computing equipment
[0092] 18 surroundings
[0093] 20 images
[0094] 22 Feature Pyramid Network
[0095] 24 Feature map
[0096] Level 26
[0097] 28 Kernel Generator
[0098] 30 kernels
[0099] 32 fusion kernels
[0100] 34 high-resolution feature maps
[0101] 36 feature encoders
[0102] 38 Prediction module
[0103] 40 individual segmentation maps
[0104] 42 Object Center
[0105] 44 Background
[0106] 46 Trustworthiness
[0107] 48 individual trust values
[0108] 50 higher-level electronic computing equipment
[0109] 52 Position head element
[0110] 54 Trustworthiness
Claims
202412630 18 Patent claims 1. Method for analyzing an environment (18) using a detection device (12), comprising the steps: - Capturing an image (20) of the environment (18) by means of an optical detection device (14) of the detection device (12); - Generating a multi-level and multi-scale feature map (24) of the image (20) using an artificial intelligence of an electronic computing device (16) of the acquisition device (12); - Analyzing each level (26) of the multi-level and multi-scale feature map (24) with respect to an object center (42) in the respective level (26) and / or a background (44) in the respective level (26) using artificial intelligence; - Generating a respective kernel (30) for a respective detected object center (42) in a respective stage (26) for the feature map (24) using artificial intelligence; - Determining a respective confidence value (46) regarding the object center (42) when analyzing a respective stage (26) using artificial intelligence; - Linking a respective kernel (30) with the corresponding trust value (46) using artificial intelligence; - Merging the respective kernels (30) of a stage (26) with the respective confidence values (46) into a respective fusion kernel (32) with a confidence value (54) using artificial intelligence; and - Outputting the respective fusion kernel (32) with the associated confidence value (54) for analysis of the environment (18) using artificial intelligence.
2. Method according to claim 1, characterized in that the multi-stage and multi-scale feature map (24) is additionally analyzed by means of a feature encoder (36).
3. Method according to claim 2, characterized in that an object membership of a pixel in the image (20) is determined in a respective stage (26) depending on a respective fusion kernel (32) and depending on the analysis of the feature encoder (36).
4. Method according to claim 3, characterized in that 202412630 19 the specific object memberships of a respective level (26) for the feature map (24) are condensed and, depending on this, a single segmentation map (40) is generated.
5. Method according to claim 4, characterized in that for the individual segmentation map (40) a single confidence value (48) is determined depending on the respective confidence values (46) of the respective fusion kernels (30) and / or the kernels (30).
6. Method according to claim 4 or 5, characterized in that individual segmentation map (40) with at least the respective confidence values (46) is transmitted to a higher-level electronic computing device (50) for further processing.
7. Method according to one of the preceding claims, characterized in that, depending on a risk-based criterion, a number of kernels (30) for a respective stage (26) and a number of associated confidence values (46) are generated.
8. Method according to one of the preceding claims, characterized in that for a respective stage (26) kernels (30) which are sufficiently close to each other are fused.
9. Method according to claim 8, characterized in that a cosine similarity of the kernels (30) is used for the fusion.
10. Method according to one of the preceding claims, characterized in that a feature pyramid network (22) is used to generate the multi-level and multi-scale feature map (24).
11. Method according to one of the preceding claims, characterized in that the artificial intelligence is provided as a fully convoluted network for panoptic segmentation.
12. Method according to one of the preceding claims, characterized in that at least one hypothesis for the kernel (30) is additionally determined for a respective kernel (30). 202412630 20 13. Computer program product comprising program code means which cause an electronic computing device (16) to perform a method according to one of claims 1 to 12 when the program code means are processed by the electronic computing device (16).
14. Computer-readable storage medium comprising at least one computer program product according to claim 13.
15. Detection device (12) for analyzing an environment (18), comprising at least one optical detection device (14) and an electronic computing device (16), wherein the detection device (12) is configured to perform a method according to one of claims 1 to 12.