Method for quality assurance of a sample-based system

Abstract

The invention relates to a method for quality assurance of a sample-based system (1). In order to improve the quality assurance, in the method a sample-based system (1) is created and trained on the basis of collected samples (22), which form a sample set. The respective samples (22) of the sample set comprise input values (12), which lie in an input space (20). From the distribution of the input values (12) in the input space (20) a quality assessment is determined (C), which represents the coverage of the input space (20) by the samples (22) of the sample set.

Description

[0001] This invention relates to a method for quality assurance of sample-based systems.

[0002] Example-based systems, such as artificial neural networks, are known in principle. They are typically used in domains where no direct algorithmic solution exists or where they cannot be properly created using traditional software methods. Example-based systems allow task settings to be created and trained on multiple examples. The learned task settings can then be applied to multiple other examples.

[0003] The development of feedforward artificial neural networks with supervised learning is described in Thomas Waschulzik's paper "Efficient development of quality assurance for feedforward artificial neural networks with supervised learning (QUEEN)" (hereinafter referred to as: WASCHULZIK).

[0004] In this context, the technical problem to be solved by the present invention is to improve the quality assurance of sample-based systems.

[0005] According to the present invention, the above-mentioned technical problem is solved by a method for quality assurance of a sample-based system, in which a sample-based system is created and trained based on collected samples forming a sample set. Each sample in the sample set includes input values ​​located in an input space. A quality assessment (or quality metric) is determined based on the distribution of the input values ​​in the input space, the quality assessment representing the coverage of the input space by the samples in the sample set.

[0006] This invention is based on the understanding that example-based systems, such as neural networks, are generally considered black boxes. Therefore, internal information processing is not analyzed, and the generation of an understandable model is omitted. Furthermore, the system is not validated through inspection. This leads to the retention of example-based systems when used in high-critical task settings.

[0007] This invention is also based on the understanding that when collecting samples to create and train sample-based systems, it is usually unknown how many samples must be collected in which regions of the input space to create a suitable knowledge base.

[0008] The solution according to the invention eliminates these problems by determining the coverage of the input space by examples based on the distribution of input values ​​within the input space. This yields a mapping of the input space, which serves as the basis for further example collection to create a suitable knowledge base. Therefore, example collection can be controlled according to the distribution in the input space, although the specific type of classifier or approximator is not yet determined. Nor is it necessary to determine the degrees of freedom for training the knowledge base. By understanding which regions must be further sampled, examples can be collected in a more targeted manner, and thus the cost of example collection can be significantly reduced (because fewer examples must be collected overall).

[0009] This invention also recognizes that for example-based systems, a prerequisite for using a mapping of the input space is the appropriate representation and encoding of features. Raw data is transformed into a representation adapted to the task setting through application-specific transformations. This representation is then transformed using standard methods so that it can be used as input to the activity of neurons in a neural network (so-called encoding). The quality assessment of the coverage of the input space by the examples representing the example set can be used at both the representational and encoding levels.

[0010] This invention is also based on the understanding that the encoding and / or representation of input features in the input space preferably has a semantic association with the expected output of the example-based system. For example, pixel values ​​of an RGB image are not suitable as input for classifications that are invariant to object size, rotation, and translation. For example, if features that have a semantic association with the output are determined through preprocessing, then the input space is preferably mapped.

[0011] The present invention is also based on the understanding that the ratio between the number of independent input features that determine the dimensions of the state space to be spanned and the number of samples to be collected for the configuration, training, evaluation and testing of the system should preferably not be too large: because in the case of a large ratio, the samples do not cover the input space sufficiently.

[0012] The present invention is also based on the understanding that dimensions spanning the state space are preferably semantically independent of each other (i.e., representing independent aspects of the task). More preferably, the dimensions have the same relevance to the solution for the task setting.

[0013] More preferably, for quality assurance, only a single classification task or approximation task is considered. For example, in an artificial neural network used as a Single Shot Multibox Detector (SSD), only classification of a preset object size is considered in the so-called default box (i.e., having a preset aspect ratio, a preset scaling ratio, and a preset position in the image).

[0014] Preferably, the example-based system settings are used in safety-oriented functions. Those skilled in the art will understand the term "safety-oriented function" as a safety-related function of the system, meaning its behavior affects the safety of the system's environment. Here, the term "safety" is understood in the sense of the English word "safety." In technical terms, "safety" refers to the goal of protecting the system environment from dangers emanating from the system. In contrast, in technical terms, "security" refers to the goal of protecting the system from dangers emanating from its environment.

[0015] In a preferred embodiment of the method according to the invention, the determination includes: distributing representatives in an input space and assigning a plurality of samples from a sample set to corresponding representatives. The samples assigned to a representative are located in a region of the input space surrounding that representative. A local quality assessment for the surrounding region is determined as a quality assessment.

[0016] By assigning samples from the sample set to representatives, the sample datasets assigned to the representatives are determined within the surrounding region. Local quality assessments are then calculated for each of these sample datasets.

[0017] Dividing the sample set into multiple surrounding regions offers advantages typically derived from divide-and-conquer methods in informatics. Therefore, developers of sample-based systems can focus, for example, on portions of the input space where measured quality assessments do not meet specific quality criteria. In these portions, quality can be examined accordingly and improved if necessary. This significantly reduces the cost of evaluating the entire sample set.

[0018] Preferably, representative samples are distributed as representatives. This distribution is preferably uniform. Here, for example, a grid for arranging the representative samples is selected in the input space. The grid can be determined individually for each dimension of the input space. For example, in categorical variables, the criterion used to determine the grid could be a model of the target attribute of the sample distribution in the input space, set based on the requirements of the sample-based system. The grid can be constructed hierarchically to, for example, map hierarchical encoding. When a grid is applied to arrange representative samples, the representative samples are distributed in each hypercube in the input space of the grid. In the hierarchical construction of the grid, representative samples are distributed at each hierarchical level.

[0019] Alternatively, the representative is the center of the cluster determined by a clustering method. The clustering method is preferably used to determine the location and extent of the corresponding cluster in the input space. More preferably, the clustering method is performed considering the output values ​​of the samples located in the output space. Clusters can be determined based on requirements for the properties of the sample-based system or based on a subset of the sample data. In applications of sample-based systems, for example, a sample set can be collected early on, selected based on knowledge used to meet the requirements. Therefore, this distribution of the sample data is quality-assured. In subsequent project phases, further samples can be collected using the same distribution. In this case, each sample in the quality-assured sample set represents a representative for subsequent sample collection phases. This ensures an additional quality-assured sample set for each initial sample collection. The location of the representative can be determined, for example, by the cluster center. Alternatively, a hierarchical clustering method can be used, in which each cluster and each hierarchical level is inserted with a representative, and in this hierarchical clustering method, each sample in each hierarchical level belongs to a cluster and therefore belongs to a representative. The set of samples that can be used to calculate the quality assessment is then assigned to clusters and therefore to representatives using a preset metric. For samples that cannot be assigned to a cluster, a new cluster with representatives is preferably created. Alternatively, the sample is collected separately for quality assessment along with other samples that cannot be assigned to a cluster.

[0020] More preferably, these samples are not fully assigned to a single representative, but rather to a predetermined share. This can be achieved, for example, by using a clustering algorithm that partially assigns samples to a sample dataset (e.g., percentage assignments for multiple surrounding regions, where the sum of shares is 1). When determining the quality assessment based on this partial assignment, the corresponding samples are considered according to the relevant share.

[0021] The quality assessment is preferably determined based on the number of samples assigned to the corresponding representative or based on other characteristics. This is particularly advantageous if specific samples are no longer used in further steps. Alternatively or additionally, specific samples or references to samples are stored in the representative (converting the sample dataset into a structure oriented towards the input space). This is advantageous if specific samples are needed in further steps.

[0022] Preferably, the storage space required for processing is reduced by storing a representative only if at least one sample exists in the corresponding surrounding area. When measuring coverage of the input space, surrounding areas where no representative is created are evaluated as "no sample". However, a histogram can be created with respect to the number of samples per representative, because the number of surrounding areas where no sample was collected can be determined with low cost (total expected representatives - created representatives = number of fields where no sample was detected).

[0023] According to another preferred embodiment of the method according to the invention, the quality assessment includes statistical means, which are determined based on a sample set and / or samples belonging to the corresponding representatives.

[0024] In this way, information associated with representatives can be used, for example, descriptive statistics (as described in one of the following textbooks: "Statistik: Der Weg zur Datenanalyse" (Springer textbook), paperback, September 15, 2016, by Ludwig Fahrmeir, Christian Heumann, Rita Künstler, Iris Pigeot, and Gerhard Tutz; "Statistik für Dummies" paperback, December 4, 2019, by Deborah J. Rumsey, Beate Majetschak (translator), and Reinhard Engel (translator); "Arbeitsbuch zur deskriptiven und induktiven Statistik" (Springer textbook), paperback, February 27, 2009, by Helge Toutenburg (author), Michael Schomaker (contributor), Malte Wißmann (contributor), and Christian Heumann). Heumann (the contributor) used methods to define quality assessment.

[0025] In a preferred extended design, a histogram is created as a statistical measure to represent the number of samples assigned to the representative.

[0026] This enables a particularly simple and intuitive way to evaluate and display coverage of the input space.

[0027] Those skilled in the art will preferably understand the expression "regarding the number of samples assigned to the representative" as meaning that the value of the number of samples assigned to the representative is binned (i.e. divided into regions) for the purpose of creating a histogram.

[0028] According to another preferred extended design, statistical measures, particularly the mean, median, minimum, maximum, and / or quantiles of the number of representative samples, are determined as statistical means.

[0029] According to another preferred extended design, adjacent surrounding areas are measured in the input space, and their corresponding representatives are assigned multiple samples that meet the preset quality standards for quality assessment.

[0030] If the number of samples assigned to the corresponding representative is lower than or higher than the preset quality threshold or is within the preset quality band of the quality assessment, then the preset quality standard is preferably met.

[0031] When determining whether two surrounding regions are adjacent to each other, different neighborhood relations can be used, such as von Neumann neighborhood (also known as 4-neighborhood), Moore neighborhood (also known as 8-neighborhood), or neighborhood in graph theory. The defined neighborhood relations must be transferred accordingly in higher-dimensional spaces: thus, in three-dimensional space, consider, for example, the 6-neighborhood of a cube with a common face, the 18-neighborhood of a cube with a common edge, and the 26-neighborhood of a cuboid with a common corner point. Here, a neighborhood is defined by how many dimensions two grid points are allowed to differ in order to still be considered adjacent.

[0032] In a preferred extended design, an associated region is determined within the input space, the associated region being composed of adjacent surrounding regions, each of which is assigned a plurality of samples that meet a preset quality standard.

[0033] If the number of samples assigned to the corresponding representative is lower than or higher than the preset quality threshold or is within the preset quality band of the quality assessment, then the preset quality standard is preferably met.

[0034] If the quality standard is met by passing below a preset quality threshold, the location and size of areas in the input space where too few samples are collected (essentially "holes" in the input space) can be determined in a particularly advantageous manner. In other words, a particular advantage of this implementation is that it identifies regions in the input space where sample values ​​do not provide a sufficient basis for safety-critical applications. This, in turn, allows for corrective interventions, such as collecting further samples or restricting the knowledge base in the application to regions with high-quality relevance.

[0035] Measurements of regions with too few collected samples are particularly advantageous, as they offer preventative protection against adversarial attacks. This is because the success rate of adversarial attacks is relatively high in these regions. This success rate can be reduced by collecting more samples in these areas or by limiting the knowledge base to regions with high-quality associations.

[0036] Quality assessments can be calculated based on the determined association regions. Therefore, for example, the number of representatives within an association region can be determined. Histograms of the size or other attributes of the association regions can be created. Furthermore, statistical measures of the attributes of the association regions, such as mean, median, quantiles, or standard deviation, can be calculated. Additionally, the extent of the association regions along the input spatial dimensions can be determined. Dimensions can be ordered according to the maximum extent of the association regions.

[0037] According to another preferred embodiment of the method according to the invention, if the quality assessment measured for the corresponding surrounding area is less than a preset quality threshold, further samples are collected in the corresponding surrounding area. Alternatively or additionally, if the quality assessment measured for the corresponding surrounding area is greater than the preset quality threshold, samples are removed from the corresponding surrounding area.

[0038] According to a particularly preferred embodiment of the method according to the invention, the corresponding sample includes an output value located in the output space. A local complexity assessment is determined for the corresponding surrounding region, which represents the complexity of the task setting of the sample-based system defined by the samples in the surrounding region. The local complexity assessment is determined by the relative positions of the samples in the surrounding region with respect to each other in the input and output spaces.

[0039] Those skilled in the art will preferably understand "the relative positions of the samples in the surrounding area with respect to each other in the input and output spaces" as defining the complexity assessment based on considerations of the similarity between the distances of the samples in the input space and their distances in the output space. For example, if the distances in the input space (except for scaling) roughly correspond to the distances in the output space, then the task setup of a sample-based system has relatively low complexity.

[0040] This yields the advantage of efficient sample collection. Because the complexity assessment reveals regions where a relatively large number of samples must be collected due to the high complexity of the task setting of the sample-based system. Preferably, in regions of the input space with high complexity, the density of representatives is dynamically increased until a uniform complexity is achieved and a sufficient number of samples are located in the surrounding environment of the representatives.

[0041] Complexity assessments correspond, for example, to the quality metrics described in Section 4 (QUEEN Quality Metrics) of WASCHULZIK. These quality metrics can be defined and applied for the representation or encoding of a feature (see Section 4.5 of WASCHULZIK).

[0042] According to a preferred embodiment of the method according to the invention, the integration quality index QI based on Section 4.6 of WASCHULZIK is used. 2 As a representative quality indicator, this integrated quality indicator is defined according to Formula 4.21 as follows:

[0043]

[0044] Among them, according to WASCHULZIK's formula 4.18:

[0045]

[0046] It is the normalized distance of the input (NRE) it represents, and

[0047]

[0048] This is the normalized distance of the represented output (NRA). Here, x is a pair (x1, x2) consisting of two samples x1 and x2. x1 and x2 are samples from the sample set P. P = {p1, p2, ..., p...} |p|} is the set of elements of BAG P, where |P| is the number of elements in BAG P. BAG is a multiset (also called a bag) defined in specification 21.5 on page 27 of the WASCHULZIK appendix. The task setting QAG is defined in definition 3.1 on page 23 of the WASCHULZIK and is referred to there as the QUEEN task setting.

[0049] d RE (x) is the input space d re (vep xl ,vep x2 d is an abbreviation for medium distance. RA (x) is the output space d ra (vap xl ,vap x2 (This is an abbreviation for medium distance.)

[0050] According to WASCHULZIK's definition of the distance between representatives of two samples, it is based on the Euclidean norm. Therefore, the distance in the input space is defined as (see WASCHULZIK's Equation 4.3):

[0051]

[0052] p k1 p k2 As an example of set P, where

[0053]

[0054] in,

[0055] i is the running index for all expressions;

[0056] vemp i,kx Let be the representation of the input feature i of the sample kx, where kx∈R (R is the set of real numbers); and

[0057] aem is the number of input features in the QAG (Quick Ability Group) setting.

[0058] In a preferred extended design, the complexity of the aggregation is determined by evaluating the local complexity of the aggregation.

[0059] The advantage of aggregate complexity assessment is that developers of sample-based systems can easily perform their quality assurance.

[0060] For example, a histogram of complexity in different surrounding regions of the input space is created as a complexity assessment for aggregation. To do this, the range of complexity assessment values ​​is binned (i.e., divided into regions). Preferably, when the location of a surrounding region is no longer needed, only the number of surrounding regions with the corresponding complexity is included in the bin. Preferably, the histogram is combined with information about the number of samples, for example, also combined into a histogram about the number of samples assigned to a representative. More preferably, information about the representatives is stored in the histogram so that this information can be traced during detailed analysis.

[0061] According to another preferred extended design, surrounding regions whose complexity assessments are below a preset complexity threshold are identified based on the complexity evaluation of the aggregate. Within these identified surrounding regions, task settings for the example-based system are implemented using an algorithmic solution. This is particularly advantageous for applications with high-quality requirements, such as in security-oriented functions.

[0062] This preferred extension design is based on the understanding that the exact way the system operates (i.e., semantic associations) is generally known for regions with low task setup complexity. In this case, the task setup can be implemented as a conventional algorithm (rather than a sample-based system). This is particularly advantageous because it is generally easier to prove the sufficient security of a safety-oriented function within the scope of approval procedures for simple algorithmic solutions.

[0063] This extended design also yields the advantage of eliminating the need to collect further samples in areas of low complexity.

[0064] When searching simple regions, it is preferable to also search for data collection artifacts, which give a relationship between input and output that is specific to the data collection process but does not represent a practically usable relationship (e.g., known from the so-called Kluger-Hans effect: https: / / de.wikipedia.org / wiki / Kluger_Hans). In regions with particularly high complexity, samples are analyzed in this way to see if, for example, problems arise in the collection and acquisition of samples.

[0065] According to another preferred embodiment of the method according to the invention, the input space is divided according to a quality assessment hierarchy.

[0066] Preferably, hierarchical mapping of the input space is achieved through hierarchical partitioning of the input space. More preferably, the hierarchical structure is derived from the representation or encoding of input features and / or from the complexity analysis of the task setting.

[0067] By introducing additional hierarchical structures into the analysis of the input space, the density of representations can be dynamically increased in regions with high complexity (until uniform complexity is achieved) or new hierarchical levels can be introduced. New hierarchical levels are introduced by adding new partitions with higher resolution to the represented regions. This process can be iterated by adding additional hierarchical levels in high-resolution regions where local complexity increases again. This allows the resolution to be dynamically adapted to the appropriate task settings.

[0068] According to another preferred embodiment of the method according to the invention, the complexity distribution is determined by means of a histogram representation of the complexity evaluation of the k nearest neighbors of the sample in the input space. In this way, how the complexity is distributed for the local surrounding environment of the sample is determined. Thus, the characteristics of the complexity in the local surrounding environment of the sample are determined, and it can be said that the fingerprint of the local surrounding environment of the sample in terms of complexity is determined.

[0069] For histogram representations, it is preferable to bin (i.e. divide) the range of values ​​for the complexity assessment. For example, the binned values ​​are plotted on the y-axis, and the representation of the increment k (k nearest neighbors) is recorded on the x-axis.

[0070] To reduce the computational power required to determine the complexity distribution, a step size greater than 1 is chosen for the value of k. For example, for values ​​of k=5, 10, 15, 20, etc., the distribution of the complexity evaluation is determined with a step size of 5. More preferably, the step size of k is chosen to be smaller only in regions of particular interest. Thus, the distribution of the complexity evaluation is calculated, for example, first with a relatively large step size of k, so that it can then be calculated with a smaller step size of k in regions of particular interest.

[0071] More preferably, the number of complexity evaluation values ​​is stored for each calculated histogram field (complexity evaluation binned, k). More preferably, identification information of the sample (e.g., number) is also stored, with the complexity distribution determined in the sample's surrounding environment.

[0072] According to another preferred embodiment of the method according to the invention, a sample-based system setting is used in a safety-oriented function, wherein the safety-oriented function includes object recognition based on image recognition, in which the sample-based system is used to identify objects.

[0073] In a preferred extended design, object recognition is used in the automated operation of transportation vehicles, particularly rail vehicles, motor vehicles, aircraft, watercraft and / or spacecraft.

[0074] Object recognition in the automated operation of transportation vehicles is a particularly suitable implementation of safety guidance functions. Here, for example, object recognition is needed to identify obstacles on the travel path or to analyze traffic conditions regarding the right-of-way of road users.

[0075] Motor vehicles include cars, such as sedans (PKW), trucks (LKW), or tracked vehicles.

[0076] Water transport vehicles include, for example, ships or submarines.

[0077] The means of transportation can be manned or unmanned.

[0078] An example of this application is the autonomous or automated driving of rail vehicles. To address the task setting, object recognition systems are used to analyze the scene, which is digitized using sensors. For example, scene analysis is needed to identify obstacles on the driving path or to analyze traffic conditions regarding right-of-way for road users. To identify objects, systems based on usage examples are currently being used with particular success, utilizing these examples to train the parameters of the pattern recognition system. An example of this is, for instance, a neural network with a deep learning algorithm.

[0079] According to another preferred embodiment of the method according to the invention, a sample-based system setting is used in a safety-oriented function, wherein the safety-oriented function includes classification based on sensor data from the organism.

[0080] Tissue classification of animal or human tissues is a particularly suitable implementation of safety-guided functions in the field of medical image processing. Organisms include, for example, archaea (protozoa), bacteria (true bacteria), and eukaryotes (nuclear) or tissues derived from protozoa (also known as Protoctista, after the founder), Plantae (plants), Fungi (mushrooms, chitinous fungi), and Animals (animals).

[0081] Further applications include the safety control of industrial equipment (e.g., synthesis in chemistry, control of manufacturing processes such as steel rolling), the classification of chemical substances (e.g., environmental toxins, war agents), the classification of vehicle signatures (e.g., radar or ultrasonic signatures), and / or control in the field of industrial automation (e.g., machine manufacturing).

[0082] According to another preferred embodiment of the method according to the invention, the sample-based system includes

[0083] - Systems with supervised learning capabilities

[0084] - Systems constructed using statistical methods,

[0085] - Preferably, an artificial neural network with one or more layers of neurons, which are neither input nor output neurons and are trained using backpropagation.

[0086] - Especially convolutional neural networks,

[0087] - Especially single-shot multi-box detector networks.

[0088] Using artificial neural networks can often improve classification or approximation performance.

[0089] One or more layers of neurons that are neither input nor output neurons are often technically referred to as "hidden" neurons. Training a neural network with multiple layers of hidden neurons is also technically referred to as deep learning. A particular type of deep learning network used for pattern recognition is the so-called Convolutional Neural Network (CNN). A special case of CNN is the so-called SSD (Single Shot MultiBox) network. Those skilled in the art will understand the term "Single Shot MultiBox Detector" as a method for object recognition based on a deep learning approach that is grounded in a convolutional neural network and described in: Liu Wei (SSD paper published October 2016: Single Shot MultiBox Detector. European Conference on Computer Vision. Lectures in Computer Science. 9905. pp. 21-37. arXiv:1512.02325).

[0090] The present invention also relates to a computer program including instructions that, when implemented by a computing unit, cause the computing unit to perform a method of the type described above.

[0091] The present invention also relates to a computer-readable storage medium including instructions that, when implemented by a computing unit, cause the computing unit to perform a method of the type described above.

[0092] Advantages, implementation methods, and details of the features of the computer program and computer-readable storage medium according to the invention can be found in the above description of the corresponding features of the method according to the invention.

[0093] Embodiments of the present invention are illustrated with reference to the accompanying drawings. In the drawings:

[0094] Figure 1 The flowchart of an embodiment of the method according to the present invention is illustrated schematically.

[0095] Figure 2 The schematic diagram illustrates the structure of a sample-based system according to an embodiment of the method of the present invention.

[0096] Figure 3 A two-dimensional input space according to an embodiment of the method according to the invention is schematically shown.

[0097] Figure 4 A schematic side view of a rail vehicle located on a roadway is shown.

[0098] Figure 5 The hierarchical partitioning of the input space is shown.

[0099] Figure 6 The diagram shows two axis plots representing the application of complexity evaluation to the first composition function.

[0100] Figure 7 Two axis plots representing the application of complexity evaluation to the second composition function are shown.

[0101] Figure 8 The diagram shows two axis plots representing the application of complexity evaluation to the third composition function, and

[0102] Figure 9 This schematically illustrates another example of a two-dimensional input space according to another embodiment of the method according to the invention.

[0103] Figure 1 A schematic flowchart is shown, which represents an embodiment of a method for quality assurance of a sample-based system according to the present invention.

[0104] Figure 2 The structure of a sample-based system 1 is schematically illustrated, in which system quality assurance is performed through an embodiment of the method according to the invention. The sample-based system 1 is a supervised learning system formed by an artificial neural network 2 having a layer 4 of input neurons 5 and a layer 6 of output neurons 7. The artificial neural network 2 has multiple layers 8 of neurons 9, which are neither input neurons 5 nor output neurons 7. The artificial neural network 2 is a so-called multilayer perceptron, but it can also be a recurrent neural network, a convolutional neural network, or, in particular, a so-called single-shot multi-box detector network.

[0105] The example-based system and the method according to the invention are implemented using one or more computer programs. The computer program includes instructions that, when implemented by a computing unit, cause the computing unit to perform operations according to... Figure 1 The illustrated embodiment is based on the method of the present invention. The computer program is stored on a computer-readable storage medium.

[0106] The sample-based system is used in the system's security-oriented functions. Therefore, the behavior of the function affects the security of the system environment.

[0107] An example of a safety-oriented function is image recognition-based object recognition, in which objects are identified using a sample-based system 1. Object recognition is used, for example, in transportation vehicles, especially... Figure 4 The automated operation of rail vehicles 40, motor vehicles, airplanes, watercraft or spacecraft shown in the figure.

[0108] Other examples of safety-oriented functions are classification of sensor data from organisms, such as archaea (protozoa), bacteria (true bacteria), and eukaryotes (nuclears), or from protozoa (also known as Protoctista, Plantae, Fungi, and Animals), for safety control of industrial equipment, classification of chemical substances, classification of vehicle signatures, or control in the field of industrial automation.

[0109] In step A of the method, it is determined which samples to collect. In step B, samples are collected: the collected samples form a sample set. Each sample has an input value 12 in the input space and an output value 14 in the output space. (In the context of...) Figure 4 In the case of object recognition during the automated operation of the illustrated rail vehicle 40 (as one of several possible examples of a safety guidance function), examples are collected by a camera unit 42 provided on the rail vehicle 40 for acquiring images. The camera unit 42 is oriented in the direction of travel 41 such that it captures a spatial region 43 located in front of the direction of travel 41. The rail vehicle 40 travels along a travel segment 44 in the direction of travel 41 together with the camera unit 42. To collect examples, the scene associated with the creation and training of the example-based system 1 for object recognition is reenacted. Thus, for example, cardboard cutouts, crash test dummies, or actors 45 are used to represent people on the travel segment 44, which should be identified by the example-based system 1 to be created and trained. Alternatively, the scene can be reenacted using so-called virtual reality.

[0110] In method step C, a quality assessment is determined, which represents the coverage of the input space by the samples in the sample set. When determining the quality assessment in C, the samples in method step C1 will be represented by their distribution in the input space. Figure 3 A two-dimensional input space 20 is shown as an example. In practical applications of the method according to the invention, the input and output spaces often have higher dimensions. Example 22 of the sample set is shown below. Figure 3The crosshairs 23 are shown in the center. 24 represent the evenly distributed points 25 of the grid shown in the diagram 26.

[0111] In method step C2, multiple samples 29 from the sample set are assigned to corresponding representatives 28. The samples 29 assigned to representatives 28 are located in a surrounding region 30 of the input space 20, surrounding the corresponding representative 28. The surrounding region 30 is exemplarily located in… Figure 3 The area is represented as a dotted region. Here, in method step C3, the local quality assessment of the surrounding area 30 is determined as the quality assessment.

[0112] In method step C4, adjacent surrounding regions 32-36 are measured in the input space, representing the number of samples with a quality threshold below a preset value. Figure 3 In the middle, the surrounding area 32-36 is represented as the region with diagonal stripes. Figure 3 In the example shown, surrounding regions 32-36 are areas in which no samples are found. Furthermore, in method step C5, a correlated region 38 is determined within the input space 20. This correlated region 38 consists of adjacent surrounding regions 32-36, each of which is associated with a number of samples below a preset quality threshold. This determines the location and size of areas in the input space 20 where too few samples have been collected. In other words, it identifies regions of the input space where the sample values ​​do not provide a sufficient basis for safety-critical applications.

[0113] Based on this identification, corrective interventions can be implemented: for example, in method step D, if the quality assessment measured for the corresponding surrounding area is less than a preset quality threshold, further samples are collected in the corresponding surrounding area.

[0114] In method step E, a local complexity assessment is determined for the corresponding surrounding region. This local complexity assessment represents the complexity of the task setting of the sample-based system defined by the samples in the surrounding region. Here, according to method step E1, the local complexity assessment is determined by the relative positions of the samples in the surrounding region with each other in the input space 20 and the output space. That is, the complexity assessment is defined based on consideration of the similarity between the distances of the samples in the input space 20 and their distances in the output space. For example, if the distances in the input space 20 (except for scaling) roughly correspond to the distances in the output space, the task setting of the sample-based system has relatively low complexity. The complexity assessment determines the region where a relatively large number of samples must be collected due to the high complexity of the task setting of the sample-based system. For example, in regions of the input space 20 where high complexity exists, the density of representation is dynamically increased until a uniform complexity is reached. Alternatively, a new hierarchical level can be introduced (see below). Figure 5(As exemplarily described).

[0115] Complexity assessment corresponds to the quality metrics described in Section 4 (QUEEN Quality Metrics) of WASCHULZIK. Quality metrics can be defined and applied for the representation or encoding of a feature (see Section 4.5 of WASCHULZIK). An example of a quality metric used for representation is the integration quality metric QI according to Section 4.6 of WASCHULZIK. 2 .

[0116] In method step E2, the complexity assessment of the aggregation is determined by aggregating the local complexity assessment: for example, a histogram of complexity in different surrounding regions of the input space is created as the complexity assessment of the aggregation. To do this, the range of complexity assessment values ​​is binned (i.e., divided into regions). When the location of a surrounding region is no longer needed, only the number of surrounding regions with the corresponding complexity is included in the bin. This histogram is then combined with information about the number of samples, for example, also combined into a histogram about the number of samples assigned to representatives. More preferably, information about representatives is stored in the histogram so that this information can be traced during detailed analysis.

[0117] In method step F, a complexity assessment can be performed to determine if an appropriate number of samples have been collected in all regions. If a region is identified as having collected too many low-complexity samples, these samples can be removed from that region. Reducing the number of samples decreases the storage space requirements and costs for computations, such as for quality assurance measures based on the amount of sample data. If a region is identified as having collected too few samples (e.g., due to relatively high complexity), further samples must be collected in that region if necessary. The latter often occurs in regions where new hierarchical levels have been introduced (see below). Figure 5 (As exemplarily described). After collecting further samples, the loop used for quality assurance (according to method steps C to E) is iterated until all expected quality requirements are met.

[0118] In method step G, surrounding regions whose complexity assessments are below a preset complexity threshold are identified based on the complexity evaluation of the aggregation. If the system's operation (i.e., semantic associations) is known for the surrounding regions, the task setting of the example-based system is implemented in the determined surrounding regions using an algorithmic solution according to method step H. Therefore, the system's task setting is implemented as a conventional algorithm (rather than an example-based system). For regions in the input space where a statistical system or neural network should be used, a statistical system is also created or the structure of a neural network is determined and the neural network is trained in step H.

[0119] Figure 5An exemplary illustration shows the hierarchical partitioning of input space 120, through which hierarchical mapping of the input space is achieved. The collected sample 122 in the sample set... Figure 5 The images are displayed as pentagram 123 and circle 125. Pentagram 123 and circle 125 are examples of different object categories (i.e., they have different positions in the output space).

[0120] A new hierarchical level 126 can be additionally introduced into regions with high complexity. For example, a new hierarchical level 126 can be introduced by adding a new partition 132 with a higher resolution 134 to region 130. The process can be iterated by adding additional hierarchical levels in high-resolution regions where local complexity increases again.

[0121] To understand the properties and behavior of the quality metrics described in WASCHULZIK as examples of complexity assessment, it is helpful to apply the quality metrics to composition functions (e.g., y=x). From this, one can infer how these quality metrics can be applied in example-based systems.

[0122] Figures 6 to 8 Histograms of the complexity evaluation distribution of the k nearest neighbors of a pre-selected sample are shown for each composition function. The sample is, for example, the center of a representative sample or cluster (as described above). The sample can also be selected from the surrounding region of the representative sample, chosen to further investigate the complexity of the task setting.

[0123] Figure 6 Image 4.1 of WASCHULZIK is shown on the left and image 4.4 is shown on the right. As a composition function, y = x in... Figure 6 The left side shows the axis chart (entries in the axis chart are indicated by "+"). The right side shows the axis chart for the function y=x, QI 2 SHLQ 2 The histogram of the k nearest neighbors of the example is shown. The example is illustrated as the SHLQ of the histogram for any local surrounding environment k of the example. 2 The value is zero.

[0124] Figure 7 Image 4.17 of WASCHULZIK is shown on the left and image 4.20 of WASCHULZIK is shown on the right. As a composition function, y = ru(seed, 300) * 300... Figure 7 The left side is shown as an axis plot. It represents a uniformly distributed random variable taking values ​​between 0 and 300. The right side of the axis plot shows the QI for the function y = ru(seed, 300) * 300. 2 SHLQ 2Histogram of k-nearest neighborhoods of the samples. Figure 7 The axis plot on the right is scaled such that 40 represents the value 1.

[0125] Figure 8 Image 4.41 of WASCHULZIK is shown on the left and image 4.44 of WASCHULZIK is shown on the right. As a synthetic function, y = sin (8*pi*x / 300) + br (seed,300) is used in Figure 8 shown as an axis plot on the left. It is a sine function with random noise in the ranges of 0<x≤50 and 100<x≤200. The axis plot on the right shows for the function y = sin (8*pi*x / 300) + br (seed,-300), QI 2 of SHLQ 2 Histogram of k-nearest neighborhoods of the samples. Figure 8 the axis plot in is scaled such that 40 represents the value 1. Those skilled in the art will find in these diagrams that there are multiple k-neighborhoods with a size of up to about 45, where QI 2 the value of is almost 0 (represented by the dark gray shadow of the bins with small numbers plotted on the V axis), thus there is an almost linear mapping between the input space and the output space. If those skilled in the art analyze in which surrounding area of a sample a low complexity exists by reading the information in the histogram, a sample at x=75 is obtained, in whose neighborhood k=45 the complexity is very low. For k=45, this also applies to x=225 or x=275. Therefore, those skilled in the art can identify regions with particularly low or particularly high complexity simply, quickly and reliably, without prior knowledge of situations such as how the samples are distributed in the input space. By reading the bins with high values, regions with high complexity can be identified even in large surrounding areas (for example, bin number 80, K=20). This identification of regions with high or low complexity can be performed independently of the dimensions of the input and output spaces, because the distances between k-nearest neighborhoods can be determined in spaces of any dimension. Through a similar process, those skilled in the art can also identify, for example, representatives that contain very few samples from the histogram about the size of the associated region. The positions in the input space where further samples must be collected can then be determined by the representatives.

[0126] As an alternative to the embodiment described with reference Figure 3 to (according to Figure 3 which the representatives are uniformly distributed in the input space), Figure 9 an embodiment of the input space 220 is shown, in which the representatives each form the centers of clusters determined by means of a clustering method. The samples 222 of the sample set are Figure 9 shown as crosshairs 223 in the figure.

[0127] Figure 9 Four clusters, 230, 232, 234, and 236, are illustrated, each comprising multiple samples. These samples are located within dashed boundary lines in the illustration; however, these dashed boundary lines do not represent the actual boundaries of the clusters and are drawn for illustrative purposes only. Clusters 230, 232, 234, and 236 have associated cluster centers 240, 242, 244, and 246 (shown as plus signs), respectively. The cluster centers 240, 242, 244, and 246 are centrally located within the cluster and belong to the cluster independently of the boundaries of the input space's grid.

[0128] according to Figure 9 Clusters have the advantage that they represent the topological structure of data in a particularly suitable way. According to Figure 3 The grid method has the advantage of more appropriately mapping uncovered regions. For example, (according to method step C) the coverage of the input space can be computed using the grid, and the complexity evaluation (according to method step E) can be computed not only using the grid but also using cluster centers. Which approach is more suitable also depends on the neural network approach. If the encoding neurons can move in the input space, a clustering scheme is preferred, or the cluster centers can be equated to the positions of the encoding neurons in the input space.

Claims

1. A method for quality assurance of a sample-based system (1) used in object recognition or biological classification based on image recognition, wherein, Images of the spatial region ahead of the vehicle's camera unit are acquired. The scene is re-enacted to create and train a sample-based system for object recognition. The acquired image data is converted into a representative of the solution adapted to the task setting. This representative is then transformed using standard methods, making it usable as input neurons for a neural network, thus forming the input values ​​for the sample. - The sample-based system (1) is created and trained based on the collected samples (22), which form a sample set. - The corresponding sample (22) of the sample set includes an input value (12), which is located in an input space (20), which is a multidimensional space in which the sample input value is located. - A quality assessment determination (C) based on the distribution of the input values ​​(12) in the input space (20), the quality assessment representing the coverage of the input space (20) by the samples (22) of the sample set, the quality assessment determination (C) comprising: - The distribution (C1) of (24, 28) is represented in the input space (20), and - Assign (C2) multiple samples (29) of the sample set to the corresponding representatives (28), The example (29) associated with the representative (28) is located in the surrounding area (30) of the input space (20) surrounding the representative (29), and the local quality assessment determination (C3) for the surrounding area (30) is the quality assessment.

2. The method according to claim 1, in, The quality assessment includes statistical methods, based on... - The sample set and / or - Examples assigned to the corresponding representative (29) To determine the statistical methods mentioned above.

3. The method according to claim 2, in, A histogram is created to represent the number of samples that are assigned to the representative, as a statistical method.

4. The method according to claim 2, in, The determination of statistical measures is used as the statistical method.

5. The method according to claim 1 or 2, in, If the quality assessment measured for the corresponding surrounding area (32-36) is less than the preset quality threshold, then (D) further samples are collected in the corresponding surrounding area (32-36), and / or in which, If the quality assessment measured for the corresponding surrounding area is greater than the preset quality threshold, the sample is removed from the corresponding surrounding area (30).

6. The method according to claim 1 or 2, wherein, - Corresponding examples include output values ​​(14) located in the output space. - For the corresponding surrounding region determination (E), a local complexity assessment is performed, the local complexity assessment representing the complexity of the task setting of the sample-based system (1) defined by samples of the surrounding region, and - The local complexity assessment (E1) is determined by the relative positions of the surrounding region with respect to each other in the input space (20) and output space, wherein the complexity assessment is an integrated quality index (QI). 2 , - The integrated quality indicators are defined as follows: - in: It is the normalized distance of the input it represents, and It represents the normalized distance of the output. - Where x is a pair (x1, x2) consisting of two samples x1 and x2. - Where x1 and x2 are samples from sample set P, - Where P={p1,p2,...,p |p| } is the set of elements of the plural set BAG P, and - Where |P| is the number of elements in the tuple BAG P.

7. The method according to claim 6, in, The complexity assessment of (E2) aggregation is determined by aggregating the local complexity assessment.

8. The method according to claim 7, in, The complexity assessment of the aggregated region identifies the region surrounding (G), where the complexity assessment of the surrounding region is lower than a preset complexity threshold, and In the measured surrounding area, the task settings of the sample-based system described in (H) are achieved through an algorithmic solution.

9. The method according to claim 1 or 2, in, The input space (20, 120) is divided according to the quality assessment grading.

10. The method according to claim 6, in, The complexity distribution is determined using a histogram representation of the complexity evaluation of the k nearest neighbors of the sample in the input space.

11. The method according to claim 4, in, The statistical measures are the mean, median, minimum, maximum and / or quantiles of the number of samples belonging to the representative sample.

12. A computer program product, wherein when the program is implemented by a computing unit, the program causes the computing unit to perform the method according to any one of claims 1 to 11.

13. A computer-readable storage medium comprising a program, which, when implemented by a computing unit, causes the computing unit to perform the method according to any one of claims 1 to 11.

Images