Method for quality assurance of a sample-based system

Abstract

The invention relates to a method for quality assurance of a sample-based system (1). In order to improve the quality assurance, in the method a sample-based system (1) is created and trained on the basis of collected samples (22), which form a sample set. The respective samples (22) of the sample set comprise input values (12) in an input space (20). A first sample group (21) comprising a plurality of samples and a second sample group comprising a plurality of samples are collected (B1, D1). A first quality assessment is determined (C1) on the basis of the distribution of the input values in the input space, which represents the coverage of the input space by the samples of the first sample group. A second quality assessment is determined (D2) on the basis of the distribution of the input values in the input space, which represents the coverage of the input space by the samples of the second sample group. The first quality assessment and the second quality assessment are compared with each other.

Description

[0001] This invention relates to a method for quality assurance of a sample-based system.

[0002] Example-based systems, such as artificial neural networks, are known in principle. They are typically used in domains where no direct algorithmic solution exists or where they cannot be properly created using traditional software methods. Example-based systems allow task settings to be created and trained on multiple examples. The learned task settings can then be applied to multiple other examples.

[0003] In Thomas Waschulzik's paper " effiziente The development of feedforward artificial neural networks with supervised learning is described in "Künstlicher Neuronaler Netze mitüberwachtem Lernen (QUEEN) (Efficient Development of Feedforward Artificial Neural Networks with Quality Assurance with Supervised Learning (QUEEN))" (hereinafter referred to as: WASCHULZIK).

[0004] In this context, the technical problem to be solved by the present invention is to improve the quality assurance of sample-based systems.

[0005] According to the present invention, the above-mentioned technical problem is solved by a method for quality assurance of a sample-based system, in which a sample-based system is created and trained based on collected samples forming a sample set. Each sample in the sample set includes an input value located in an input space. A first sample set comprising multiple samples and a second sample set comprising multiple samples are collected. A first quality assessment (or quality index) is determined based on the distribution of input values ​​in the input space, the first quality assessment representing the coverage of the input space by the samples of the first sample set. A second quality assessment (or quality index) is determined based on the distribution of input values ​​in the input space, the second quality assessment representing the coverage of the input space by the samples of the second sample set. The first quality assessment and the second quality assessment are compared with each other.

[0006] This invention is based on the understanding that example-based systems, such as neural networks, are generally considered black boxes. Therefore, internal information processing is not analyzed, and the generation of an understandable model is omitted. Furthermore, the system is not validated through inspection. This leads to the retention of example-based systems when used in high-critical task settings.

[0007] This invention is also based on the understanding that when collecting samples to create and train sample-based systems, it is usually unknown how many samples must be collected in which regions of the input space to create a suitable knowledge base.

[0008] Another fundamental understanding of this invention is that samples for sample-based systems are typically acquired in multiple steps. During this iterative process, it is desirable to ensure that no application-related differences arise between the acquisition of the first and second sample groups, or at least that these differences can be identified and evaluated.

[0009] An example of a technical condition is the change in the characteristics of the sensor used when collecting samples due to aging. In this technical condition, the difference between the first sample group and the second sample group is relevant: the sensor calibration has changed due to aging, and it must be checked whether the first sample group collected at the first degree of sensor aging and the second sample group collected at the second degree of sensor aging still describe the same task settings.

[0010] Another example of different cases is that one sample set was deliberately tampered with to harm the creators or users of the knowledge base.

[0011] The solution according to the invention eliminates these problems by determining a first quality assessment and a second quality assessment and comparing the two quality assessments. The results may, for example, indicate that one of the sample data sets must be discarded, either entirely or partially.

[0012] The corresponding quality assessment represents the coverage of the input space by the examples, which is determined based on the distribution of input values ​​within the input space. This enables the mapping of the input space. This mapping can be used as the basis for further example acquisition to create a suitable knowledge base. Therefore, example acquisition can be controlled according to the distribution in the input space, although the specific type of classifier or approximator has not yet been determined. Nor is it necessary to determine the degrees of freedom for training the knowledge base. By understanding which regions must be further sampled, examples can be acquired in a more targeted manner, and thus the cost of sample acquisition can be significantly reduced (because fewer examples must be acquired overall).

[0013] This invention also recognizes that for sample-based systems, a prerequisite for mapping the input space is the appropriate representation and encoding of features. Raw data is transformed into a representative of the solution adapted to the task setting through application-specific transformations. This representation is then transformed using standard methods so that it can be used as input to the activity of neurons in a neural network (so-called encoding). The quality assessment of the coverage of the input space by the representative sample set can be used at both the representational and encoding levels.

[0014] The present invention is also based on the understanding that the encoding and / or representation of input features in the input space preferably has a semantic association with the expected output of the sample-based system. For example, if features that have a semantic association with the output are determined through preprocessing, then the input space is preferably mapped.

[0015] The present invention is also based on the understanding that the ratio between the number of independent input features that determine the dimensions of the defined state space and the number of samples to be collected for the configuration, training, evaluation and testing of the system should preferably not be too large: because in the case of a large ratio, the samples do not cover the input space sufficiently.

[0016] The present invention is also based on the understanding that the dimensions defining the state space are preferably semantically independent (i.e., representing independent aspects of the task). More preferably, the dimensions have the same relevance to the solution for the task setting.

[0017] More preferably, for quality assurance, only a single classification task or approximation task is considered. For example, in an artificial neural network used as a Single Shot Multibox Detector (SSD), only classification of a preset object size is considered in the so-called default box (i.e., having a preset aspect ratio, a preset scaling ratio, and a preset position in the image).

[0018] Preferably, the example-based system settings are used in safety-oriented functions. Those skilled in the art will understand the term "safety-oriented function" as a safety-related function of the system, meaning its behavior affects the safety of the system's environment. Here, the term "safety" is understood in the sense of the English word "safety." In technical terms, "safety" refers to the goal of protecting the system environment from dangers emanating from the system. In contrast, in technical terms, "security" refers to the goal of protecting the system from dangers emanating from its environment.

[0019] Another example of the application of the method according to the invention is a situation in which the first sample group has already undergone quality assurance and the associated first quality assessment is known. If the quality assessment of the second sample group is corresponding or similar, then the quality of the second sample group can be considered satisfactory.

[0020] According to a preferred embodiment of the method according to the invention, a third sample group is formed from the first and second sample groups, and a third quality assessment is determined based on the distribution of input values ​​in the input space, the third quality assessment representing the coverage of the input space by the samples of the third sample data group. The first quality assessment, the second quality assessment, and the third quality assessment are compared.

[0021] The third sample set can be said to represent the union of the first and second sample sets.

[0022] An example of the application of the third sample set is the following situation: in this situation, a second sample set is collected given the knowledge gained from the first sample set. For example, the second sample set is collected to fill gaps in the input space. The third sample set can be used to determine whether the gaps have been successfully filled.

[0023] In a preferred embodiment of the method according to the invention, the determination of the quality assessment includes: distributing representatives in an input space and assigning a plurality of samples from a sample set to corresponding representatives. The samples assigned to a representative are located in a region of the input space surrounding that representative. A first quality assessment is determined based on the samples assigned to representatives from a first sample set for the surrounding region. A second quality assessment is determined based on the samples assigned to representatives from a second sample set for the surrounding region.

[0024] By assigning samples from the sample set to representatives, the sample datasets assigned to the representatives are determined within the surrounding region. Local quality assessments, namely the first quality assessment and the second quality assessment, are then calculated for each sample dataset.

[0025] For example, it might be of interest to further investigate those surrounding areas where the relative differences between the first and second quality assessments are relatively large.

[0026] Dividing the sample set into multiple surrounding regions offers advantages typically derived from the divide-and-conquer approach in computer science. Therefore, developers of sample-based systems can, for example, focus on the portions of the input space that do not meet specific quality criteria. Within these portions, quality can be examined accordingly and improved if necessary. This significantly reduces the cost of evaluating the entire sample set.

[0027] Preferably, representative samples are distributed as representatives. This distribution is preferably uniform. Here, for example, a grid for arranging the representative samples is selected in the input space. The grid can be determined individually for each dimension of the input space. For example, in categorical variables, the criterion used to determine the grid could be a model of the target attribute of the sample distribution in the input space, set based on the requirements of the sample-based system. The grid can be constructed hierarchically to, for example, map hierarchical encoding. When a grid is applied to arrange representative samples, the representative samples are distributed in each hypercube in the input space of the grid. In the hierarchical construction of the grid, representative samples are distributed at each hierarchical level.

[0028] Alternatively, the representative is the center of the cluster determined by a clustering method. The clustering method is preferably used to determine the location and extent of the corresponding cluster in the input space. More preferably, the clustering method is performed considering the output values ​​of the samples located in the output space. Clusters can be determined based on requirements for the properties of the sample-based system or based on a subset of the sample data. In applications of sample-based systems, for example, a sample set can be collected early on, selected based on knowledge used to meet the requirements. Therefore, this distribution of the sample data is quality-assured. In subsequent project phases, further samples can be collected using the same distribution. In this case, each sample in the quality-assured sample set represents a representative for subsequent sample collection phases. This ensures an additional quality-assured sample set for each initial sample collection. The location of the representative can be determined, for example, by the cluster center. Alternatively, a hierarchical clustering method can be used, in which each cluster and each hierarchical level is inserted with a representative, and in this hierarchical clustering method, each sample in each hierarchical level belongs to a cluster and therefore belongs to a representative. The set of samples that can be used to calculate the quality assessment is then assigned to clusters and therefore to representatives using a preset metric. For samples that cannot be assigned to a cluster, a new cluster with representatives is preferably created. Alternatively, the sample is collected separately for quality assessment along with other samples that cannot be assigned to a cluster.

[0029] More preferably, these samples are not fully assigned to a single representative, but rather to a predetermined share. This can be achieved, for example, by using a clustering algorithm that partially assigns samples to a sample dataset (e.g., percentage assignments for multiple surrounding regions, where the sum of shares is 1). When determining the quality assessment based on this partial assignment, the corresponding samples are considered according to the relevant share.

[0030] The quality assessment is preferably determined based on the number of samples assigned to the corresponding representative or based on other characteristics. This is particularly advantageous if specific samples are no longer used in further developments. The comparison parameter used to compare the first and second quality assessments can be the magnitude of the difference between the number of samples in the first sample group assigned to the representative and the number of samples in the second sample group assigned to the representative.

[0031] Alternatively or additionally, specific examples or references to examples are stored in the representation (converting the sample dataset into a structure oriented towards the input space). This is advantageous if specific examples are needed in further directions.

[0032] Preferably, the storage space required for processing is reduced by storing a representative only if at least one sample exists in the corresponding surrounding area. When measuring coverage of the input space, surrounding areas where no representative is created are evaluated as "no sample". However, a histogram can be created with respect to the number of samples per representative, because the number of surrounding areas where no sample was collected can be determined with low cost (total expected representatives - created representatives = number of fields where no sample was detected).

[0033] In a preferred extension, the third quality assessment is a local quality assessment of the surrounding area based on the samples assigned to the representative from the third sample group.

[0034] Preferably, when determining the first, second, and / or third quality assessments, representatives are distributed to the same locations in the input space. In other words, the locations of the representatives are preferably selected to be the same across all sample groups. This achieves the mutual assignment of representatives among the first, second, and / or third sample groups.

[0035] According to another preferred embodiment of the method according to the invention, the quality assessment includes statistical means, which are determined based on a sample set and / or samples belonging to the corresponding representatives.

[0036] In this way, information associated with representatives can be used, for example, descriptive statistics (as described in one of the following textbooks: "Statistik: Der Weg zur Datenanalyse" (Springer textbook), paperback, September 15, 2016, by Ludwig Fahrmeir, Christian Heumann, Rita Künstler, Iris Pigeot, and Gerhard Tutz; "Statistik für Dummies" paperback, December 4, 2019, by Deborah J. Rumsey, Beate Majetschak (translator), and Reinhard Engel (translator); "Arbeitsbuch zur deskriptiven und induktiven Statistik" (Springer textbook), paperback, February 27, 2009, by Helge Toutenburg (author), Michael Schomaker (contributor), Malte Wiβmann (contributor), and Christian Heumann). Heumann (the contributor) used methods to define quality assessment.

[0037] Histograms can be used to categorize and process the measured local quality assessments. For example, the relative difference between the first quality assessment (for the first sample group) and the second quality assessment (for the second sample group) can be measured for each surrounding area. A histogram is then created based on this, binning the samples according to their relative differences. The histogram representation further allows, for example, the selection of those surrounding areas already assigned to a bin by clicking on the corresponding bin. One of these surrounding areas can then be selected, and additional information, such as the distribution of samples, can be displayed in detail.

[0038] According to another preferred extended design, statistical measures, particularly the mean, median, minimum, maximum, and / or quantiles of the number of representative samples, are determined as statistical means.

[0039] According to another preferred extended design, adjacent surrounding areas are measured in the input space, and their corresponding representatives are assigned a certain number of samples, which meet the preset quality standards for quality assessment.

[0040] If the number of samples assigned to the corresponding representative is lower than or higher than the preset quality threshold or is within the preset quality band of the quality assessment, then the preset quality standard is preferably met.

[0041] When determining whether two surrounding regions are adjacent to each other, different neighborhood relations can be used, such as von Neumann neighborhood (also known as 4-neighborhood), Moore neighborhood (also known as 8-neighborhood), or neighborhood in graph theory. The defined neighborhood relations must be transferred accordingly in higher-dimensional spaces: thus, in three-dimensional space, consider, for example, the 6-neighborhood of a cube with a common face, the 18-neighborhood of a cube with a common edge, and the 26-neighborhood of a cuboid with a common corner point. Here, a neighborhood is defined by how many dimensions two grid points are allowed to differ in order to still be considered adjacent.

[0042] In a preferred extended design, a correlated region is determined within the input space, which consists of adjacent surrounding regions. Representatives of each surrounding region are assigned a certain number of samples. This number meets a preset quality standard.

[0043] If the number of samples assigned to the corresponding representative is lower than or higher than a preset quality threshold or falls within a preset quality band for quality assessment, then the preset quality standard is preferably met. More preferably, if the quality assessment measured for the corresponding surrounding area is lower than the preset quality threshold, then further samples are collected in the corresponding surrounding area. Alternatively or additionally, if the quality assessment measured for the corresponding surrounding area is higher than the preset quality threshold, then samples are removed from the corresponding surrounding area.

[0044] If the quality standard is met by passing below a preset quality threshold, the location and size of areas in the input space where too few samples are collected (essentially "holes" in the input space) can be determined in a particularly advantageous manner. In other words, a particular advantage of this implementation is that it identifies regions in the input space where sample values ​​do not provide a sufficient basis for safety-critical applications. This, in turn, allows for corrective interventions, such as collecting further samples (e.g., as a second set of samples described above) or restricting the knowledge base in the application to regions with high-quality relevance.

[0045] Measurements of regions with too few collected samples are particularly advantageous, as they offer preventative protection against adversarial attacks. This is because the success rate of adversarial attacks is relatively high in these regions. This success rate can be reduced by collecting more samples in these areas or by limiting the knowledge base to regions with high-quality associations.

[0046] Quality assessments can be calculated based on the determined association regions. Therefore, for example, the number of representatives within an association region can be determined. Histograms of the size or other attributes of the association regions can be created. Furthermore, statistical measures of the association region's attributes, such as mean, median, quantiles, or standard deviation, can be calculated. Additionally, the extent of the association region along the input spatial dimensions can be determined. Dimensions can be ordered according to the maximum extent of the association regions.

[0047] In the example of the application of the third sample set described above, determining the correlation region is particularly advantageous, where a second sample set is collected given the knowledge gained from the first sample set. In this example, the second sample set is collected to fill gaps in the input space. The gaps can be identified and characterized based on the aforementioned correlation region. Whether the gaps have been successfully filled can be determined based on the third sample set.

[0048] According to a particularly preferred embodiment of the method according to the invention, the corresponding sample includes an output value located in the output space. A local complexity assessment is determined for the corresponding surrounding region, which represents the complexity of the task setting of the sample-based system as defined by the samples in the surrounding region. The local complexity assessment is determined by the relative positions of the samples in the surrounding region with respect to each other in the input and output spaces.

[0049] Those skilled in the art will preferably understand "the relative positions of the samples in the surrounding area with respect to each other in the input and output spaces" as defining the complexity assessment based on considerations of the similarity between the distances of the samples in the input space and their distances in the output space. For example, if the distances in the input space (except for scaling) roughly correspond to the distances in the output space, then the task setup of a sample-based system has relatively low complexity.

[0050] This yields the advantage of efficient sample collection. Because the complexity assessment reveals regions where a relatively large number of samples must be collected due to the high complexity of the task setting of the sample-based system. Preferably, in regions of the input space with high complexity, the density of representatives is dynamically increased until a uniform complexity is achieved and a sufficient number of samples are located in the surrounding environment of the representatives.

[0051] Complexity assessments correspond, for example, to the quality metrics described in Section 4 (QUEEN Quality Metrics) of WASCHULZIK. These quality metrics can be defined and applied for the representation or encoding of a feature (see Section 4.5 of WASCHULZIK).

[0052] In a particularly preferred embodiment of the method according to the invention, a first complexity assessment is determined for samples from a first set of samples, a second complexity assessment is determined for samples from a second set of samples, and a third quality assessment is determined for samples from a third set of samples. The third local complexity assessment is compared with the first and / or second local complexity assessments.

[0053] Pay particular attention to the following surrounding area: In this area, the third complexity assessment shows a significantly higher complexity for the third sample group compared to the first and / or second complexity assessments. In this area, either there is a problem with the sample collection, or the first, second, or both sample data groups have been tampered with to compromise the knowledge base creators or users. This application example is particularly important because high complexity cannot be detected solely from the first and second sample groups (but only from the third sample group).

[0054] For the situation described above, one possible approach is to remove a sample from one of the two sample groups and determine whether an "improvement" (i.e., lower complexity) has been achieved based on a recalculated complexity assessment. If an improvement has been achieved, further samples can be removed. This process can proceed if a third complexity assessment shows a drift towards lower complexity.

[0055] To address the corresponding differences between the first, second, and third complexity assessments, an acceptance range or degree of acceptance can be defined. In other words, if the consistency falls within a specific range, the collected samples are accepted. The degree of acceptance can be chosen based on the size of the dataset. For relatively large datasets, a narrower acceptance range is preferable, while for relatively small datasets, a wider acceptance range is preferable.

[0056] In another preferred embodiment of the method according to the invention, the complexity distribution is determined by means of a histogram representation of complexity evaluation.

[0057] Preferably, the histogram representation bins (i.e. divides) the range of values ​​for complexity evaluation into regions.

[0058] In a preferred extension, the complexity distribution of a sample over its k nearest neighbors in the input space is determined. In this way, the complexity distribution is determined for the local environment of the sample. Specifically, the characteristics of the complexity in the local environment of the sample are determined, and it can be said that the fingerprint of the local environment of the sample in terms of complexity is determined.

[0059] For example, the "bin" value is plotted on the y-axis, and the representation of the increased k (k nearest neighbor) is entered on the x-axis.

[0060] To reduce the computational power required to determine the complexity distribution, a step size greater than 1 is chosen for the value of k. For example, for values ​​of k = 5, 10, 15, 20, etc., the distribution of the complexity evaluation is determined with a step size of 5. More preferably, the step size of k is chosen to be smaller only in regions of particular interest. Thus, the distribution of the complexity evaluation is calculated, for example, first with a relatively large step size of k, so that it can then be calculated with a smaller step size of k in regions of particular interest.

[0061] More preferably, the number of complexity evaluation values ​​is stored for each calculated histogram field (complexity evaluation binned, k). More preferably, identification information of the sample (e.g., number) is also stored, with the complexity distribution determined in the sample's surrounding environment.

[0062] According to a preferred embodiment of the method according to the invention, the integration quality index QI based on Section 4.6 of WASCHULZIK is... 2 The integrated quality indicator used as a representative quality indicator can be defined based on Formula 4.21 as follows:

[0063]

[0064] Among them, according to WASCHULZIK's formula 4.18:

[0065]

[0066] It is the normalized distance of the input (NRE) it represents, and

[0067]

[0068] It is the normalized distance of the represented output (NRA). Here, x is a pair (x1, x2, ..., p1) consisting of two samples x1 and x2. x1 and x2 are samples from the sample set P. P = {p1, p2, ..., p1 ... |p|} is the set of elements of BAG P, where |P 2 | represents the number of elements in BAG P. BAG is a multiset (or bag) defined in specification 21.5 on page 27 of the WASCHULZIK appendix. The task setting QAG is defined in definition 3.1 on page 23 of the WASCHULZIK and is referred to there as the QUEEN task setting.

[0069] d RE (x) is the input space d re (vep x1 ,vep x2 d is an abbreviation for medium distance. RA (x) is the output space d ra (vap xl ,vap x2 (This is an abbreviation for medium distance.)

[0070] According to WASCHULZIK's definition of the distance between representatives of two samples, which is based on the Euclidean norm, the distance in the input space is defined as (see WASCHULZIK's Equation 4.3):

[0071]

[0072] p k1 p k2 As an example of set P, where

[0073]

[0074] in,

[0075] i is the running index for all expressions;

[0076] vemp i,kx is the representation of the input feature i of the sample kx, where kx∈R (R is the set of real numbers); and aem is the AnzahlEingabeMerkmale (number of input features) of the task setting QAG.

[0077] Those skilled in the art will preferably understand the phrase "based on the following definition" or "can be defined based on Formula 4.21 as" meaning the quality index F(QI). 2 Modifications and functions are contained within the ideas of this definition.

[0078] In a preferred extended design, the complexity of the aggregation is determined by evaluating the local complexity of the aggregation.

[0079] The advantage of aggregate complexity assessment is that developers of sample-based systems can easily perform their quality assurance.

[0080] For example, a histogram of complexity in different surrounding regions of the input space is created as a complexity assessment for aggregation. To do this, the range of complexity assessment values ​​is binned (i.e., divided into regions). Preferably, when the location of a surrounding region is no longer needed, only the number of surrounding regions with the corresponding complexity is included in the bin. Preferably, the histogram is combined with information about the number of samples, for example, also combined into a histogram about the number of samples assigned to a representative. More preferably, information about the representatives is stored in the histogram so that this information can be traced during detailed analysis.

[0081] According to another preferred extended design, surrounding regions whose complexity assessments are below a preset complexity threshold are identified based on the complexity evaluation of the aggregate. Within these identified surrounding regions, task settings for the example-based system are implemented using an algorithmic solution. This is particularly advantageous for applications with high-quality requirements, such as in security-oriented functions.

[0082] This preferred extension design is based on the understanding that the exact way the system operates (i.e., semantic associations) is generally known for regions with low task setup complexity. In this case, the task setup can be implemented as a conventional algorithm (rather than a sample-based system). This is particularly advantageous because it is generally easier to prove the sufficient security of a safety-oriented function within the scope of approval procedures for simple algorithmic solutions.

[0083] This extended design also yields the advantage of eliminating the need to collect further samples in areas of low complexity.

[0084] When searching simple regions, it is preferable to also search for data collection artifacts, which give a relationship between input and output that is specific to the data collection process but does not represent a practically usable relationship (e.g., known from the so-called Kluger-Hans effect: https: / / de.wikipedia.org / wiki / Kluger_Hans). In regions with particularly high complexity, analyze whether problems arise in the collection and acquisition of samples.

[0085] According to another preferred embodiment of the method according to the invention, the input space is divided according to a quality assessment hierarchy.

[0086] Preferably, hierarchical mapping of the input space is achieved through hierarchical partitioning of the input space. More preferably, the hierarchical structure is derived from the representation or encoding of input features and / or from the complexity analysis of the task setting.

[0087] By introducing additional hierarchical structures into the analysis of the input space, the density of representations can be dynamically increased in regions with high complexity (until uniform complexity is achieved) or new hierarchical levels can be introduced. New hierarchical levels are introduced by adding new partitions with higher resolution to the represented regions. This process can be iterated by adding additional hierarchical levels in high-resolution regions where local complexity increases again. This allows the resolution to be dynamically adapted to the appropriate task settings.

[0088] According to another preferred embodiment of the method according to the invention, a sample-based system setting is used in a safety-oriented function, wherein the safety-oriented function includes object recognition based on image recognition, in which the sample-based system is used to identify objects.

[0089] In a preferred extended design, object recognition is used in the automated operation of transportation vehicles, particularly rail vehicles, motor vehicles, aircraft, watercraft and / or spacecraft.

[0090] Object recognition in the automated operation of transportation vehicles is a particularly suitable implementation of safety guidance functions. Here, for example, object recognition is needed to identify obstacles on the travel path or to analyze traffic conditions regarding the right-of-way of road users.

[0091] Motor vehicles include cars, such as sedans (PKW), trucks (LKW), or tracked vehicles.

[0092] Water transport vehicles include, for example, ships or submarines.

[0093] The means of transportation can be manned or unmanned.

[0094] An example of this application is the autonomous or automated driving of rail vehicles. To address the task setting, object recognition systems are used to analyze the scene, which is digitized using sensors. For example, scene analysis is needed to identify obstacles on the driving path or to analyze traffic conditions regarding right-of-way for road users. To identify objects, systems based on usage examples are currently being used with particular success, utilizing these examples to train the parameters of the pattern recognition system. An example of this is, for instance, a neural network with a deep learning algorithm.

[0095] According to another preferred embodiment of the method according to the invention, a sample-based system setting is used in a safety-oriented function, wherein the safety-oriented function includes classification based on sensor data from the organism.

[0096] Tissue classification of animal or human tissues is a particularly suitable implementation of safety-guided functions in the field of medical image processing. Organisms include, for example, archaea (protozoa), bacteria (true bacteria), and eukaryotes (nuclear) or tissues derived from protozoa (also known as Protoctista, after the founder), Plantae (plants), Fungi (mushrooms, chitinous fungi), and Animals (animals).

[0097] Further applications include the safety control of industrial equipment (e.g., synthesis in chemistry, control of manufacturing processes such as steel rolling), the classification of chemical substances (e.g., environmental toxins, war agents), the classification of vehicle signatures (e.g., radar or ultrasonic signatures), and / or control in the field of industrial automation (e.g., machine manufacturing).

[0098] According to another preferred embodiment of the method according to the invention, the example-based system includes a system with supervised learning.

[0099] - A system constructed using statistical methods,

[0100] - Preferably, an artificial neural network with one or more layers of neurons, which are neither input nor output neurons and are trained using backpropagation.

[0101] -Especially convolutional neural networks,

[0102] - Especially single-shot multi-box detector networks.

[0103] Using artificial neural networks can often improve classification or approximation performance.

[0104] One or more layers of neurons that are neither input nor output neurons are often technically referred to as "hidden" neurons. Training a neural network with multiple layers of hidden neurons is also technically referred to as deep learning. A particular type of deep learning network used for pattern recognition is the so-called Convolutional Neural Network (CNN). A special case of CNN is the so-called SSD network (Single Shot MultiBox Detector Network). Those skilled in the art will understand the term "Single Shot MultiBox Detector" as a method for object recognition based on a deep learning approach that is grounded in a convolutional neural network and described in: Liu and Wei's paper, October 2016: SSD: Single Shot Multibox Detector. European Conference on Computer Vision. Lectures in Computer Science. 9905. pp. 21-37. arXiv:1512.02325.

[0105] The present invention also relates to a computer program including instructions that, when implemented by a computing unit, cause the computing unit to perform a method of the type described above.

[0106] The present invention also relates to a computer-readable storage medium including instructions that, when implemented by a computing unit, cause the computing unit to perform a method of the type described above.

[0107] Advantages, implementation methods, and details of the features of the computer program and computer-readable storage medium according to the invention can be found in the above description of the corresponding features of the method according to the invention.

[0108] Embodiments of the present invention are illustrated with reference to the accompanying drawings. In the drawings:

[0109] Figure 1 The flowchart of an embodiment of the method according to the present invention is illustrated schematically.

[0110] Figure 2 The architecture of a sample-based system with unsupervised learning is illustrated schematically.

[0111] Figure 3 The schematic diagram illustrates the structure of a sample-based system with supervised learning according to an embodiment of the method of the present invention.

[0112] Figure 4 A two-dimensional input space according to an embodiment of the method according to the invention is schematically shown.

[0113] Figure 5 The illustration schematically shows the state in the second phase. Figure 4 The two-dimensional input space shown

[0114] Figure 6 A schematic side view of a rail vehicle located on a roadway is shown.

[0115] Figure 7 This schematically illustrates another example of a two-dimensional input space according to another embodiment of the method according to the invention.

[0116] Figure 8 The diagram shows two axis plots representing the application of complexity evaluation to the first composition function.

[0117] Figure 9 The diagram shows two axis plots representing the application of complexity evaluation to the second composition function, and

[0118] Figure 10 The diagram shows two axes representing the application of complexity evaluation to the third composition function.

[0119] Figure 1 A schematic flowchart is shown, representing an embodiment of a method for quality assurance of a sample-based system according to the present invention. This method can be applied to sample-based systems with both supervised and unsupervised learning.

[0120] In supervised learning, the goal is to learn a function that maps data x (as input values) to labels y. One example of supervised learning is classification, where, for example, image data x is mapped to class y (e.g., "cat"). Other examples of supervised learning include regression, object recognition, image annotation, and so on.

[0121] In unsupervised learning, the goal is to learn the structure of data x (without using labels y). One example of unsupervised learning is clustering, where groups of data that are similar on a specific metric are identified. Other examples of unsupervised learning include dimensionality reduction or feature learning (so-called feature learning or representation learning).

[0122] Figure 2 and Figure 3 An embodiment of system 1 based on the example is shown. Figure 2 The structure of an example-based embodiment of System 1 is schematically illustrated, and the system is designed as an autoencoder. An autoencoder is an artificial neural network 2 that can be used for efficient data encoding and learns this ability in an unsupervised manner. The autoencoder maps input values ​​x to feature vectors Z.

[0123] Figure 3The schematic diagram illustrates the structure of an embodiment of a sample-based system 1 with supervised learning, which is designed as a multilayer perceptron. Other examples of systems with supervised learning may be recurrent neural networks, convolutional neural networks, or, in particular, the so-called Single-Shot-MultiBox-Detector-Netz.

[0124] The example-based system 1 is formed by an artificial neural network 2, which has an input neuron 5 of layer 4 and an output neuron 7 of layer 6.

[0125] exist Figure 3 The artificial neural network 2 shown has multiple layers 8 of neurons 9, which are neither input neurons 5 nor output neurons 7.

[0126] The example-based system and the method according to the invention are implemented using one or more computer programs. The computer program includes instructions that, when implemented by a computing unit, cause the computing unit to perform operations according to... Figure 1 The illustrated embodiment is based on the method of the present invention. The computer program is stored on a computer-readable storage medium.

[0127] Example-based systems are used in safety-oriented functions of a system. Therefore, the behavior of these functions affects the safety of the system environment. An example of a safety-oriented function is image recognition-based object recognition, in which objects are identified using an example-based system 1 (under supervised learning). Object recognition is used, for example, in transportation vehicles, especially... Figure 4 The automated operation of rail vehicles 40, motor vehicles, airplanes, watercraft or spacecraft shown in the figure.

[0128] Other examples of safety-oriented functions are classification of sensor data from organisms, such as archaea (protozoa), bacteria (true bacteria), and eukaryotes (nuclears), or from protozoa (also known as Protoctista, Plantae, Fungi, and Animals), for safety control of industrial equipment, classification of chemical substances, classification of vehicle signatures, or control in the field of industrial automation.

[0129] In step A of the method, it is determined which samples to collect. In step B, samples are collected: the collected samples form a sample set. Each sample has an input value 12 in the input space and an output value 14 in the output space. (In the context of...) Figure 6In the case of object recognition during the automated operation of the illustrated rail vehicle 40 (as one of several possible examples of a safety guidance function under supervised learning), examples are collected by a camera unit 42 provided on the rail vehicle 40 for acquiring images. The camera unit 42 is oriented in the direction of travel 41 such that it captures a spatial region 43 located in front of the direction of travel 41. The rail vehicle 40 travels along a travel segment 44 in the direction of travel 41 together with the camera unit 42. In order to collect examples, the scene associated with the creation and training of the example-based system 1 for object recognition is reenacted. Thus, for example, cardboard cutouts, crash test dummies, or actors 45 are used to represent people on the travel segment 44, which should be identified by the example-based system 1 to be created and trained. Alternatively, the scene can be reenacted by means of so-called virtual reality.

[0130] Figure 4 An example of a two-dimensional input space 20 is shown. In practical applications of the method according to the invention, the input and output spaces often have higher dimensions.

[0131] According to the method of the present invention, in method step B1, a first sample group 21 comprising a plurality of samples 22 is collected. The samples 22 in sample group 21 are... Figure 4 It is displayed as a crosshair 23.

[0132] In method step C1, a first quality assessment is determined, which represents the coverage of the input space by the samples of the first sample group 21. In the determination of the quality assessment in C1, the representatives are distributed in the input space in method step C2. The representatives 24 are uniformly distributed and represented as the intersections 25 of the grid 26 shown.

[0133] In method step C3, multiple samples 29 from the sample set are assigned to corresponding representatives 28. The samples 29 assigned to representatives 28 are located in a surrounding region 30 of the input space 20 surrounding the corresponding representative 28. The surrounding region 30 is exemplarily located in… Figure 4 The area is represented as a point-like surface. Here, in method step C4, a first local quality assessment for the surrounding region 30 is determined as a quality assessment. Furthermore, for each... Figure 4 Other surrounding areas shown in the figure were used to determine local quality assessments.

[0134] In method step C5, adjacent surrounding regions 32-36 are measured in the input space, and their corresponding representatives are assigned a number of samples below a preset quality threshold. Figure 4 In the diagram, these surrounding regions 32-36 are represented as surfaces with diagonal stripes. Figure 4In the example shown, surrounding regions 32-36 are areas in which no samples are found. Furthermore, in method step C6, a correlated region 38 is determined within the input space 20. This correlated region 38 consists of adjacent surrounding regions 32-36, each of which is associated with a number of samples below a preset quality threshold. This determines the location and size of areas in the input space 20 where too few samples have been collected. In other words, it identifies regions of the input space where the sample values ​​do not provide a sufficient basis for safety-critical applications.

[0135] Based on this identification, corrective interventions can be implemented: for this purpose, for example, in further method step D1, samples from the second sample group 27 are collected, and the understanding of the first sample group is taken into account during the collection. The samples from the second sample group 27 are... Figure 5 It is represented as a star shape.

[0136] Corresponding to the determination of the first local quality assessment, a second local quality assessment is determined for the surrounding area in method step D2.

[0137] The first and second sample groups together form a third sample group as a union. In method step D3, a third local quality assessment is measured for the surrounding area, which represents the coverage of the input space by the samples of the third sample group.

[0138] In method step E, the first, second, and third quality assessments are compared. For example, in method step E1, the difference between the third quality assessment and the first or second quality assessment is determined. If the quality of the sample set is improved in the case of the union compared to the first sample set 21, it can be assumed that the knowledge base has been improved through the collection of the second sample set 27.

[0139] For example, quality assessment can be determined based on the number of samples assigned to a corresponding representative. The comparison parameter used to compare the first and second quality assessments can be the magnitude of the difference between the number of samples in the first sample group assigned to the representative and the number of samples in the second sample group assigned to the representative.

[0140] In method step F1, a first local complexity assessment is determined for the corresponding surrounding region. This first local complexity assessment represents the complexity of the task setting of the sample-based system defined by a first set of samples in the surrounding region. Here, according to method step F11, the local complexity assessment is determined by the relative positions of the samples in the surrounding region relative to each other in the input space 20 and the output space. That is, the complexity assessment is defined based on consideration of the similarity between the distances of the samples in the input space 20 and their distances in the output space. For example, if the distances in the input space 20 (except for scaling) roughly correspond to the distances in the output space, the task setting of the sample-based system has relatively low complexity. The region where a relatively large number of samples must be collected due to the high complexity of the task setting of the sample-based system is determined based on the complexity assessment. For example, in regions of the input space 20 where high complexity exists, the density of representation is dynamically increased until uniform complexity is reached.

[0141] Complexity assessment corresponds to the quality metrics described in Section 4 (QUEEN Quality Metrics) of WASCHULZIK. Quality metrics can be defined and applied for the representation or encoding of a feature (see Section 4.5 of WASCHULZIK). An example of a quality metric used for representation is the integration quality metric QI according to Section 4.6 of WASCHULZIK. 2 .

[0142] In method step F2, a second local complexity assessment is determined for the corresponding surrounding region. This second local complexity assessment represents the complexity of the task setting of the sample-based system defined by the second sample set. In method step F3, a third local complexity assessment is determined for the samples in the third sample set (collection).

[0143] In method step G, the first, second, and third complexity assessments are compared. Particular attention is paid to the following surrounding areas: in these areas, the third local complexity assessment shows a significantly higher complexity for the third sample group compared to the first and / or second complexity assessments. Therefore, in these surrounding areas, either there is a problem with the sample collection, or the first, second, or both sample data groups 21 and 27 have been tampered with to compromise the creator or user of the knowledge base. This application example is particularly important because the high complexity cannot be detected solely from the first and second sample groups (but only from the third sample group).

[0144] For reference Figure 4 and Figure 5 The described alternatives represent embodiments uniformly distributed in the input space. Figure 7 An embodiment of input space 220 is shown, in which the centers represent the clusters determined by clustering methods. Example 222 of the sample set is shown. Figure 7 The middle part is represented by a crosshair 223.

[0145] Figure 7 Four clusters, 230, 232, 234, and 236, are illustrated, each comprising multiple samples. In the illustration, the samples are located within dashed boundary lines, but these dashed boundary lines do not represent the actual boundaries of the clusters and are drawn for illustrative purposes only. Clusters 230, 232, 234, and 236 have associated cluster centers 240, 242, 244, and 246 (represented by plus signs), respectively. The cluster centers 240, 242, 244, and 246 are centrally located within the cluster and belong to the cluster independently of the boundaries of the input space's grid.

[0146] according to Figure 7 Clusters have the advantage that they represent the topology of data in a particularly suitable way. According to Figure 4 and Figure 5 The grid method has the advantage that uncovered regions are mapped more appropriately. For example, the coverage of the input space can be computed using the grid (according to method steps C1-C6), and the computational complexity can be evaluated using cluster centers in addition to the grid (according to method steps F1-F3). Which method is more suitable also depends on the neural network approach. If the encoding neurons can move in the input space, a clustering method is preferred, or the cluster centers can be equated to the locations of the encoding neurons in the input space.

[0147] To understand the properties and behavior of the quality metrics described in WASCHULZIK as examples of complexity assessment, it is helpful to apply the quality metrics to composition functions (e.g., y = x). From this, one can infer how these quality metrics can be applied in example-based systems.

[0148] Figures 8 to 10 Histograms of the complexity evaluation distribution of the k nearest neighbors of a pre-selected sample are shown for each composition function. The sample is, for example, the center of a representative sample or cluster (as described above). The sample can also be selected from the surrounding region of the representative sample, chosen to further investigate the complexity of the task setting.

[0149] Figure 8 Image 4.1 of WASCHULZIK is shown on the left and image 4.4 is shown on the right. As a composition function, y = x... Figure 8 The left side is represented as an axis graph (entries in the axis graph are shown as "+"). The right side of the axis graph shows QI for the function y = x. 2 SHLQ 2 The histogram of the k nearest neighbors of the example is shown. The example is illustrated as the SHLQ of the histogram for any local surrounding environment k of the example. 2 The value is zero.

[0150] Figure 9 Image 4.17 from WASCHULZIK is shown on the left, and Image 4.20 from WASCHULZIK is shown on the right. As a synthesis function, y=ru(seed,300)*300 is Figure 9 represented as an axis diagram on the left. It is a uniformly distributed random variable with values ranging from 0 to 300. The axis diagram on the right shows, for the function y=ru(seed,300)*300, the QI 2 SHLQ 2 histogram of the k-nearest neighbor domain for the examples. Figure 9 The axis diagram on the right is scaled such that 40 represents the value 1.

[0151] Figure 10 Image 4.41 from WASCHULZIK is shown on the left, and Image 4.44 from WASCHULZIK is shown on the right. As a synthesis function, y=sin(8*pi*x / 300)+br(seed,300) is Figure 10 represented as an axis diagram on the left. It is a sine function with random noise in the ranges of 0<x≤50 and 100<x≤200. The axis diagram on the right shows, for the function y=sin(8*pi*x / 300)+br(seed,-300), the QI 2 SHLQ 2 histogram of the k-nearest neighbor domain for the examples. Figure 10 The axis diagram therein is scaled such that 40 represents the value 1. Those skilled in the art will find from these illustrations that there are multiple k-neighbor domains with a size of up to about 45, in which QI 2The value of is almost 0 (represented by the dark gray shading of bins with small numbers plotted on the V-axis), thus there is an almost linear mapping between the input and output spaces. If a person skilled in the art analyzes which sample's surrounding environment contains low complexity by reading the information in the histogram, they find that the sample x = 75 has very low complexity in its neighborhood k = 45. The same applies to x = 225 or x = 275 for k = 45. Therefore, a person skilled in the art can easily, quickly, and reliably identify regions with particularly low or high complexity without prior knowledge of how the samples are distributed in the input space. By reading the bins with high values, regions with high complexity (e.g., bin number 80, k = 20) can be identified even in large surrounding environments. This identification of regions with high or low complexity can be done independently of the dimensions of the input and output spaces, since the distance between the k nearest neighbors can be determined in any dimensional space. Through a similar process, a person skilled in the art can also identify, for example, representatives containing very few samples from a histogram of the size of the associated regions. Then, the location in the input space where further samples must be collected can be determined by the representative.

[0152] Although the invention has been described and illustrated in more detail through preferred embodiments, the invention is not limited to the disclosed examples and other variations can be derived by those skilled in the art without departing from the scope of protection of the invention.

Claims

1. A method for quality assurance of a sample-based system (1), wherein, - The sample-based system (1) is created and trained based on the collected samples, which form a sample set. - The corresponding sample in the sample set includes the input value (12), which is located in the input space (20). - Collect the first sample group (21) containing multiple samples and the second sample group containing multiple samples, (B1, D1). - A first quality assessment is determined based on the distribution of the input values ​​in the input space, whereby the first quality assessment represents the coverage of the input space by the samples of the first sample group. - A second quality assessment (D2) is determined based on the distribution of the input values ​​in the input space, the second quality assessment representing the coverage of the input space by the samples of the second sample group, and - The first quality assessment and the second quality assessment are compared with each other, wherein a comparison parameter is determined, which is a measure of the difference between the number of samples belonging to the represented first sample group and the number of samples belonging to the represented second sample group, and wherein, based on the determined comparison parameter, one of the sample groups is discarded, either entirely or partially. - Wherein, the sample-based system (1) is configured for use in a security-oriented function, and the security-oriented function includes object recognition based on image recognition, in which the sample-based system (1) is used to identify the object (45).

2. The method according to claim 1, - A third sample group is formed from the first sample group and the second sample group, and - A third quality assessment (D3) is determined based on the distribution of the input values ​​in the input space, the third quality assessment representing the coverage of the input space by the samples of the third sample group, and The first quality assessment, the second quality assessment, and the third quality assessment are compared (E).

3. The method according to claim 1 or 2, in, The quality assessment determination (C1) includes: - The representative distribution (C2) is in the input space (20), and - Assign (C3) multiple samples from the sample set to the corresponding representatives. The sample associated with the representative is located in the surrounding area (30) of the input space (20) around the representative. Among them, the local quality assessment of the surrounding area (30) based on the sample determination (C4) of the first sample group (21) belonging to the representative is used as the first quality assessment, and / or The local quality assessment of the surrounding area (30) is determined based on the sample assigned to the representative of the second sample group as a second quality assessment.

4. The method according to claim 2, in, The quality assessment determination (C1) includes: - The representative distribution (C2) is in the input space (20), and - Assign (C3) multiple samples from the sample set to the corresponding representatives. The sample associated with the representative is located in the surrounding area (30) of the input space (20) around the representative. Among them, the local quality assessment of the surrounding area (30) based on the sample determination (C4) of the first sample group (21) belonging to the representative is used as the first quality assessment, and / or The local quality assessment of the surrounding area (30) determined based on the sample assigned to the representative sample of the second sample group is used as the second quality assessment. The third quality assessment is a local quality assessment of the surrounding area (30) based on the samples assigned to the representative from the third sample group.

5. The method according to claim 3, in, The quality assessment includes statistical methods, which are based on... - The sample set and / or - The example assigned to the corresponding representative The measurements were taken.

6. The method according to claim 3, in, The quality assessment includes statistical methods, which are based on... - The sample set and / or - The example assigned to the corresponding representative To conduct measurements, Among them, the determination of statistical measures is used as the statistical method.

7. The method according to claim 3, in, In the input space (20), the adjacent surrounding areas (32-36) are measured (C5), and the corresponding representatives of the surrounding areas are assigned a certain number of samples, the number of which meets the preset quality standard of the quality assessment.

8. The method according to claim 7, in, Within the input space (20), the associated region (38) is determined (C6), which is composed of adjacent surrounding regions (32-36), and each of the surrounding regions is assigned a certain number of samples, the number of which meets the preset quality standard of the quality assessment.

9. The method according to claim 3, wherein, - The corresponding example includes the output value (14) located in the output space. - A local complexity assessment is determined for the corresponding surrounding region, the local complexity assessment representing the complexity of the task setting of the example-based system (1) defined by examples of the surrounding region, and - The local complexity evaluation is determined by the relative positions of the surrounding regions in the input space (20) and the output space relative to each other.

10. The method according to claim 2, wherein, - The corresponding example includes the output value (14) located in the output space. - A local complexity assessment is determined for the corresponding surrounding region, the local complexity assessment representing the complexity of the task setting of the example-based system (1) defined by examples of the surrounding region, and - The local complexity evaluation is determined by the relative positions of the surrounding region with respect to each other in the input space (20) and the output space. Specifically, a first local complexity evaluation (F1) is determined for the samples in the first sample group, a second local complexity evaluation (F2) is determined for the samples in the second sample group, and a third local complexity evaluation (F3) is determined for the samples in the third sample group. Specifically, the third local complexity assessment is compared with the first local complexity assessment and / or the second local complexity assessment (G).

11. The method according to claim 9, in, The complexity distribution is determined using the histogram representation of the complexity assessment.

12. The method according to claim 11, in, The complexity distribution of the sample over the k nearest neighbors in the input space is determined.

13. The method according to claim 1, in, The object recognition is used in the automated operation of transportation vehicles.

14. The method according to claim 1, in, The example-based system (1) is configured for use in safety-oriented functions, and the safety-oriented functions represent classification based on sensor data from organisms, or include safety control of industrial equipment, classification of chemical substances, classification of vehicle signatures and / or control in the field of industrial automation.

15. The method according to claim 1, in, The example-based system (1) includes - Systems with supervised learning, - An artificial neural network (2) having one or more layers (8) of neurons (9), which are neither input neurons (5) nor output neurons (7) and are trained using backpropagation.

16. The method according to claim 6, in, The statistical measures include the mean, median, minimum, maximum, and / or quantile of the number of samples associated with the representative sample.

17. The method according to claim 13, in, The means of transport include rail vehicles (40), motor vehicles, aircraft, watercraft and / or spacecraft.

18. The method according to claim 15, in, The example-based system (1) includes - Convolutional Neural Networks - Single-shot multi-box detector network.

19. A computer program product comprising instructions that, when implemented by a computing unit, cause the computing unit to perform the method according to at least one of claims 1 to 18.

20. A computer-readable storage medium comprising a computer program containing instructions that, when implemented by a computing unit, cause the computing unit to perform the method according to at least one of claims 1 to 18.

Images