Validation of computer vision models

Abstract

Validation of computer vision models. Computer vision focuses on how a computer can automatically gain high-level understanding from digital images or videos. Computer vision systems are finding increasing applications in the field of automotive or robotic vehicles. Computer vision can process input from at least one detector and any interaction between that detector and its environment. The environment can be perceived by the at least one detector as a scene or a series of scenes. In this specification, it is proposed to reduce the number of parameters in a set of vision parameters based on a sensitivity analysis of how a given vision parameter affects the performance of a computer vision model to provide a validation parameter set with a reduced size.

Description

Technical Field

[0001] This specification relates to computer-implemented methods for validating computer vision models, as well as associated apparatus, computer programs, and computer-readable media. Background Technology

[0002] Computer vision focuses on how computers can automatically gain advanced understanding from digital images or videos. There is a growing trend of applications for computer vision systems in the automotive and robotic vehicle sectors. Computer vision can process input from any interaction between at least one detector and its environment. This environment can be perceived by at least one detector as a scene or a set of scenes.

[0003] In particular, the interaction can be generated by at least one camera, a multi-camera system, a RADAR system, or a LIDAR system.

[0004] In automotive computer vision systems, computer vision often has to contend with open environments, which remain safety-critical. Therefore, considering quantitative safeguards is crucial in both the design and testing of computer vision functions. Validation of computer vision functions for autonomous driving is typically important for ensuring the safety of such functions, but the validation process requires substantial input from human experts. Therefore, the validation of computer vision functions can be further improved. Summary of the Invention

[0005] According to the first aspect, a method for validating a computer vision model is provided, comprising: - Obtain a computer vision model that is configured to perform computer vision functions that represent elements of the observed scene; - Obtain a visual parameter specification that includes at least one set of visual parameters, wherein the classification or regression performance of the computer vision model is affected by visual data items provided based on the at least one set of visual parameters; - Provide a first visual dataset, which includes a subset of visual data items conforming to the visual parameter specification and a corresponding subset of basic fact data items; - When multiple visual data items are applied to a visual dataset, perform sensitivity analysis on multiple performance scores output from a computer vision model over the domain of the visual parameter specification. - Generate validation parameter specifications based on visual parameter specifications and sensitivity analysis, wherein, compared with the visual parameter specifications, the validation parameter specifications include at least one less set of visual parameters or visual parameters, and / or wherein the validation parameter specifications include at least one set of visual parameters with a narrowed scope; - Validate the computer vision model using the following methods: - Provide a second visual dataset comprising a second subset of visual data items conforming to the validation parameter specification and a corresponding second subset of basic fact data items, wherein the validation parameter specification is used to provide the second visual dataset; and - Apply at least one validation test to the computer vision model to obtain at least one validation test result.

[0006] Advantageously, the computational load and complexity required to validate computer functions can be significantly reduced. Computer vision models can also be validated more safely because the computational load of validation can be focused on the visual parameters to which the computer vision function being validated is more sensitive. Computer vision functions are validated in an empirical and repeatable manner without the need for interpretive trade-offs among different human experts, although experts can still provide visual parameters as initial candidates during the validation process.

[0007] Computer vision functionality in autonomous systems is a safety-critical component. To ensure safety and for reasons of accountability, the validation of vision parameter specifications (sometimes referred to as the operational design domain) is combined with the development of associated computer vision functionality to provide safety assurance. Reducing the parameter space of the validation process enables faster automated validation while ensuring that the sensitive aspects of the computer vision model have been adequately validated.

[0008] Validating computer vision features using deep neural networks is a significant challenge and is crucial for their application in safety-critical systems such as autonomous vehicles and robots. How to perform validation and which attributes should be validated are difficult questions. Another challenge involves the test strategy—how test results should be structured within the overall validation plan and organized, for example, using target structured notation (GSN). Prioritization and selection of visual parameters used for validation (testing) can be used in constructing the test strategy (e.g., GSN) for the test effort and in the standardization of computer vision feature testing. In other words, the idea is to use sensitivity analysis to reduce the complexity of the test strategy.

[0009] This invention relates to a verification technique for generating test results of computer vision capabilities. These test results can be used to support decisions regarding the release of computer vision capabilities or to support autonomous systems utilizing those capabilities. This method can be combined with various testing techniques, such as search-based testing, combined testing, or statistical analysis and measurement of computer vision capabilities.

[0010] The technique can be extended by pre-selecting the attributes (visual parameters) and value ranges corresponding to the test. This reduces testing effort and increases the effectiveness of the testing method because only the relatively more relevant visual parameters are tested. In other words, it is recommended to apply global sensitivity analysis to reduce the number of parameters to be validated, while the goal of the testing technique is to apply thorough testing to a fixed set of parameters determined by the previous global sensitivity analysis. Therefore, the sample density of global sensitivity analysis is much lower than that of testing, enabling a faster validation process when validation is performed by automated hardware such as a computer.

[0011] Visual parameter specifications can be refined to include visual parameters that cause large variance in computer vision functionality. Using such visual parameter specifications to select additional images as training data can be useful for training computer vision models because the higher variance implies that images selected according to such specifications contain many "corner cases" of computer vision functionality that should be considered safe.

[0012] Visual parameter specifications can be refined to include visual parameters that cause relatively large variance in computer vision function.

[0013] Mathematically, the goal of testing is to attempt to prove that the function being tested is not working properly. When the test fails to reveal a functional malfunction, the function is validated. Therefore, particularly for validation, a subset of parameters with high variance is chosen to maximize the probability of detecting the malfunction. Additional images selected using such visual parameter specifications as training data can be useful for validating computer vision models, as higher variance suggests that images selected according to such visual parameter specifications contain extreme cases on which the computer vision function will perform even worse.

[0014] When training or validating computer vision models, reducing one or more redundant parameters in the vision parameter specification is equivalent to a significant reduction in the size of the parameter space, thereby enabling a faster and more energy-efficient training and / or validation process.

[0015] Visual parameters are any aspect of an image or video that affects a machine vision function's understanding of a displayed scene. As an example, visual parameters could be the angle of the sun relative to the vehicle, the time of day, the height and proximity of buildings and trees in the scene, the speed of the vehicle, the speed and position of vehicles relative to the vehicle, and the visual characteristics of the road surface in front of the vehicle.

[0016] As a concrete example, direct sunlight on a wet road surface in front of a vehicle can cause additional glare to shine directly onto the vehicle's windshield. This increased glare can affect how accurately a computer vision model can classify road signs located in front of the vehicle. Therefore, under the described conditions, the computer vision model will be more likely to misidentify road signs in the road sign identification task. Consequently, the performance of the computer vision model will exhibit higher variance.

[0017] The second aspect provides an apparatus for validating computer vision models, comprising: an input interface, a processor, a memory, and an output interface.

[0018] The input interface is configured to obtain a computer vision model, which is configured to perform computer vision functions representing elements of an observed scene, and to obtain a visual parameter specification including at least one set of visual parameters. The classification or regression performance of the computer vision model is affected by visual data items provided based on the at least one set of visual parameters. The input interface is configured to provide a first visual dataset, which includes a subset of visual data items conforming to the visual parameter specification and a corresponding subset of basic fact data items.

[0019] The processor is configured to perform sensitivity analysis on multiple performance scores output from a computer vision model over the domain of a visual parameter specification when multiple visual data items are applied to a visual dataset, in order to generate a validation parameter specification based on the visual parameter specification and the sensitivity analysis, wherein the validation parameter specification includes at least one less set of visual parameters or visual parameters compared to the visual parameter specification.

[0020] The validation parameter specification includes at least one set of visual parameters with a narrowed scope. The processor is further configured to validate the computer vision model by: providing a second visual dataset comprising a second subset of visual data items conforming to the validation parameter specification and a corresponding second subset of basic fact data items; providing the second visual dataset using the validation parameter specification; and applying at least one validation test to the computer vision model to obtain at least one validation test result.

[0021] The third aspect provides a computer program including machine-readable instructions that, when executed by a processor, enable the implementation of a computer-based method for verifying a computer vision model according to the first aspect.

[0022] The fourth aspect provides a computer-readable medium comprising a computer program according to the third aspect.

[0023] Dependent embodiments of the foregoing aspects are given in the dependent claims and explained in the following description, which the reader should now refer to.

[0024] This limitation applies generally to this specification.

[0025] The visual dataset of the observed scene is a collection of items representing images (such as JPEG or GIF images) or videos. Videos are sequences of images, optionally encoded in a format such as MPEG if necessary.

[0026] A computer vision model is a function parameterized by model parameters. Model parameters are learned during training using machine learning techniques based on a training dataset. A computer vision model is configured to map at least a visual data item, or a portion or subset thereof, to predicted data items. One or more visual parameters define the visual state, as they contain information about the content of the observed scene and / or represent the boundary conditions used to capture and / or generate the observed scene. The latent representation of a computer vision model is an intermediate (i.e., hidden) layer or a portion thereof within the computer vision model.

[0027] Computer vision model 16 is, for example, a deep neural network (DNN) comprising multiple neural network layers. However, other model topologies known to those skilled in the art can also be implemented according to this technique. Each layer computes a latent representation, which is a higher-level representation of the input image or video sequence.

[0028] When executed correctly, the basic fact data item corresponding to a visual data point is the classification and / or regression result that a computer vision function intends to output in response to an image input. In other words, when input along with visual data items that illustrate a predictable scene or scene elements, the basic fact data represents the correct answer from the computer vision function. The term "image" can refer to a subset of images, such as segments of road signs or obstacles.

[0029] A visual dataset of observed scenes is a collection of items representing images or videos, the latter being a sequence of images. Each item in the visual data can be a numerical tensor, with videos having additional dimensions for consecutive frames. The basic fact data item corresponding to a visual data item is, for example, the classification and / or regression result that a computer vision model should ideally output. For instance, if a visual data item is parameterized in part based on the presence of a wet road surface, and the presence or absence of a wet road surface is the intended output of the computer model being trained, then the basic facts would return a description of that item of the associated visual data as an image including the wet road surface.

[0030] Each item in the basic fact data can be another numerical tensor, or, in a simpler case, a binary result vector, which is associated with or references an item in the visual dataset. The basic fact data provides explicit results relating to scene elements in its associated visual data. For example, a visual data item including an image of a 20 km / h speed limit sign would be associated with a result vector confirming that the scene contains a road sign, with further detail that the road sign is a speed limit sign and the displayed speed limit is 20 km / h. Thus, the correctness of the output prediction of a computer vision model applied to a visual data item can be evaluated by comparing it to the associated basic fact data, and performance measures of the computer vision function can be calculated.

[0031] Visual parameter specifications, for example, define data structures that define different types of visual phenomena that can affect the performance of a computer vision function that performs functions such as image or video recognition, classification, or regression on a scene or part of a scene. Visual phenomena are encoded by one or more sets of visual parameters. For example, a visual parameter set can define parameters as a series of discrete values, such as precipitation intensity. Visual parameter sets can use Boolean conditions to define parameters, such as Boolean conditions defining the presence or absence of a tree cover. Alternatively or additionally, visual parameter sets can use continuous ranges such as azimuth displacement to define parameters.

[0032] Visual parameter specifications can be thought of as ontology, classification, dimension, or linguistic entities that represent a constrained view of the world, or as an input model. Specific collections of images can be captured (from real-world image datasets) or rendered (using a synthetic image generation engine) using visual parameters.

[0033] In embodiments, the visual parameters of the first or second visual parameter specification may include one or any combination of the following: one or more parameters describing the configuration of the image capture arrangement (optionally an image or video capture device); visual data acquired or synthesized; optionally used for spatial and / or temporal sampling; distortion aberrations; color depth; saturation; noise; surface absorption; reflectivity; one or more lighting conditions in the image / video scene; light bounce; reflection; light source; fog and light scattering; overall illumination; and / or one or more features of the image / video scene; optionally, one or more objects and / or their position, size, rotation, geometry, material, texture; one or more parameters of the environment of the image / video capture device or the simulated capture device used for synthesizing the image generator; optionally, environmental characteristics; line-of-sight; precipitation characteristics; radiant intensity; image characteristic statistics such as contrast, saturation, noise; or one or more domain-specific descriptions of the image / video scene; optionally, one or more vehicle or road users, or one or more objects at an intersection.

[0034] The figures depict exemplary embodiments of the invention, which should not be construed as limiting the claims and will be explained in more detail below. Attached Figure Description

[0035] Figure 1 A flowchart illustrating the global sensitivity analysis of computer vision capabilities is shown schematically.

[0036] Figure 2 The method implemented by a computer according to the first aspect is illustrated schematically.

[0037] Figure 3 The flowchart illustrating the verification function provided by the computer vision model is shown in the schematic diagram.

[0038] Figure 4 An example of a computer vision model validation function is illustrated schematically.

[0039] Figure 5 An example of a global sensitivity analysis function for a computer vision model is illustrated schematically.

[0040] Figure 6 The illustration shows an example of how to transform a visual parameter specification into a validation parameter specification.

[0041] Figure 7 This illustration shows an example of how a validation coverage data structure tracks the integrity of the validation process.

[0042] Figure 8 The apparatus according to the second aspect is illustrated schematically. Detailed Implementation

[0043] One or more visual parameters define the visual state of a scene because they contain information about what is being observed in the scene and / or represent the boundary conditions used to capture and / or generate the observed scene. Typical scenes are characterized by a large number of visual parameters, some of which are continuous variables. Therefore, computing a function over the total number of visual parameters characterizing a typical scene is computationally intensive.

[0044] Visual parameters can be, for example: camera characteristics (e.g., spatial and temporal sampling, distortion, aberrations, color depth, saturation, noise, etc.), LiDAR or RADAR characteristics (e.g., surface absorption or reflectivity, etc.), lighting conditions in the scene (light bounce, reflection, light sources, fog and light scattering, overall illumination, etc.), materials and textures, objects and their position, size, and rotation, geometry (of objects and the environment), parameters defining the environment, environmental characteristics such as line-of-sight, precipitation characteristics, radiance (which is suspected of strongly interacting with the detection process and may show a strong correlation with performance), image characteristics / statistics (such as contrast, saturation, noise, etc.), domain-specific descriptions of the scene and situation (e.g., cars and objects at an intersection), etc. Many more parameters are conceivable.

[0045] These parameters can be viewed as ontology, classification, dimension, or language entities. They can define a constrained view of the world or input model. Given the assignment / selection of visual parameters, a specific set of images can be captured or rendered, or visual parameters can be used to describe images in an existing dataset. The advantage of using an ontology or input model is that, for testing purposes, expected test coverage targets can be defined to define test termination criteria, such as using t-wise coverage, and for statistical analysis, distributions about these parameters can be defined.

[0046] Images, videos, and other visual data, along with other commonly labeled sensor data (GPS data, radiometric data, local meteorological characteristics), can be obtained in various ways. Real-world images or videos can be captured by image capture devices such as camera systems. Real-world images may already exist in a database, and given visual parameters and / or other sensor data, a subset of images can be manually or automatically selected. Visual parameters and / or other sensor data can also be used to define the desired experiment. Another approach is to synthesize images given visual parameters and / or other sensor data. Images can be synthesized using image augmentation techniques, deep learning networks (e.g., generative adversarial networks (GANs), variational autoencoders (VAEs)), and 3D rendering techniques. 3D rendering tools in driving simulation environments include, for example, the CARLA tool (Koltun, 2017, available at www.arXiv.org: 1711.03938).

[0047] Typically, in the development, testing, and validation of computer vision functions, input images are defined, selected, or generated based on characteristics (visual parameters) deemed important by expert opinion. However, expert opinion regarding the correct selection of visual parameters may be incomplete or misled by assumptions arising from human perceptual experience. Human perception, based on the human perceptual system (the human eye and visual cortex), differs from the technical characteristics of detection and perception using computer vision functions.

[0048] Typically, computer vision features (computer vision models) can be developed or tested on unrelated image characteristics, and visual parameters that are important influencing factors may be missed or underestimated. Furthermore, technical systems can detect additional characteristics such as extended spectral ranges or polarizations that are imperceptible to the human perceptual system.

[0049] Global sensitivity analysis has a much lower sample density than test density. This allows for prioritization and selection of visual parameters used for validation, test strategy development, or test standardization. Unlike global sensitivity analysis, test / validation then operates on a reduced test space (due to the lower number of validation parameters) and therefore executes tests at a higher test density.

[0050] Figure 1 A flowchart illustrating the global sensitivity analysis of computer vision capabilities is shown schematically. The computer vision capabilities (models) generated by this process are those validated according to this method.

[0051] World Model 10 includes visual parameters and a range or discrete values. For example, World Model 10 corresponds to the hypothetical Relevant Operational Design Domain (ODD).

[0052] Sampling engine 11 creates a data structure that includes a set of expected value combinations 12. The set of expected value combinations 12 is sampled from within the boundaries defined by world model 10. For example, sampling engine 11 can use combined tests or Latin hypercube sampling to obtain samples from world model 10.

[0053] In the example, image acquisition engine 14 passes a set of desired value combinations 12 to synthetic image generation engine 13. Optionally, synthetic image generation engine 13 is the aforementioned "CARLA" engine. This results in images, image sequences or videos, and associated underlying facts composed of values ​​sampled by sampling engine 12. Alternatively or additionally, images conforming to world model 10 can be obtained by image acquisition engine 14 querying an image database or specifying experiments to be performed.

[0054] The generated image and ground facts 15 are applied to a testing tool program 17, which applies the image to a proposed computer vision function (model) 16. Optionally, the computer vision function 16 is based on a deep neural network. The testing tool program 17 compares the predictions (regression / classification) of the computer vision function 16 with the ground facts (a portion of 15) and calculates one or more performance measures 18 by comparing the estimated image output of the computer vision function 16 with the corresponding ground facts. In other words, the testing tool evaluates how well the computer vision model classifies (or provides correct regression) the input image. For example, performance measures 18 may be, for example, confusion matrix, decoding precision, recall, F-score, intersection-over-union ratio, mean-mean ...

[0055] Sensitivity analysis 19 is performed on one or more visual parameters of world model 10 and optionally their value ranges for one or more performance measurements 18. Optionally, the value ranges are split into subranges. The result 20 is an ordered list of visual parameters associated with its performance measurement 18 when the tested computer vision function 16 is applied. Optionally, for example, the visual parameters and their subranges can be clustered 21 into regions with similar performance.

[0056] Box 1 is a practical example of a visual parameter specification. Each parenthesized text string, such as "spawn_point" or "cam_yaw," is an example of an initial set of visual parameters. In the example in Box 1, the values ​​of the initial visual parameter specification are a series of discrete values, although alternatively, at least one visual parameter specification may include a range between at least two endpoints. Alternatively, at least one set of visual parameters of a visual parameter specification may include multiple ranges (not illustrated in Box 1).

[0057] Typically, the visual parameter specifications for box 1 are defined based on expert opinions. However, experts may have incorrect opinions or be biased. In this case, the computer vision function can be validated on biased parameters.

[0058] Figure 2 A computer-implemented method according to the first aspect is schematically illustrated. The method includes: - Obtain 102 computer vision models, which are configured to perform computer vision functions that represent elements of the observed scene; - Obtain 104 visual parameter specifications including at least one set of visual parameters, wherein the classification or regression performance of the computer vision model is affected by visual data items provided based on the at least one set of visual parameters; - Provides 106 first visual datasets, which include a subset of visual data items conforming to the visual parameter specification, and a corresponding subset of basic fact data items; - When multiple visual data items are applied to a visual dataset, perform a 10⁸ sensitivity analysis on multiple performance scores output from a computer vision model over the domain of the visual parameter specification. - Generate 110 validation parameter specifications based on visual parameter specifications and sensitivity analysis, wherein, compared with the visual parameter specifications, the validation parameter specifications include at least one less set of visual parameters or visual parameters, and / or wherein the validation parameter specifications include at least one set of visual parameters with a narrowed scope; -Validate the 112 computer vision model through the following: - Provides a second visual dataset (112a) comprising a second subset of visual data items conforming to the validation parameter specification and a corresponding second subset of basic fact data items, wherein the validation parameter specification is used to provide the second visual dataset; and - Apply at least one validation test (112b) to the computer vision model to obtain at least one validation test result.

[0059] Therefore, steps 102, 104, 106, and 108 of the first aspect are similar to those described above. Figure 1 The operations described in the document.

[0060] The first aspect also includes generating validation parameter specifications. Validation parameter specifications use the results of sensitivity analysis 108 to reduce the complexity or search space of the validation process 112 for computer vision models.

[0061] The embodiment provides generating multiple predictions of elements of a scene observed in a subset of visual data items using a computer vision model, wherein the multiple predictions include at least one prediction of a classification label and / or at least one regression value of at least one item in the subset of visual data; and comparing the multiple predictions of elements in the subset of visual data items with corresponding subsets of basic fact data to obtain multiple performance scores.

[0062] In the embodiments, multiple performance scores include or depend on any one or any combination of confusion matrix, precision, recall, F1 score, crossover ratio, or mean average.

[0063] Box 2 illustrates an example of a validation parameter specification. The validation parameter specification is automatically modified according to the computer implementation method of the first aspect to identify visual parameters that cause low or average performance of the computer vision model 16. These visual parameters require additional validation and are included as high priority in the validation parameter specification.

[0064] Many different modifications can be made to the visual parameter set. For example, at least one element of the visual parameter set can be deleted, added, reduced, or added. In Box 2, the upper range of the "cloudiness" and "precipitation" parameters has been reduced from 100 to 85, and the range of the "sun altitude angle" parameter has been reduced by removing its last element. The "precipitation" parameter has been upsampled with the addition of two additional elements. The entire "cam yaw" parameter has been deleted. The above are specific examples of how the visual parameter specification changes automatically, and many changes are possible based on the results of sensitivity analysis.

[0065] As another example, the visual parameters "Sun_Altitude" and "Sun_Azimuth" form a first visual parameter cluster. The visual parameters "Cam_Yaw", "Cam_Pitch", and "Cam_Roll" form a second visual parameter cluster. The clusters can be identifiable because the sensitivity scores of each visual parameter in the first and second clusters satisfy the identification criteria.

[0066] Optionally, the identification condition is the magnitude difference between the two performance scores for "Sun_Altitude" and "Sun_Azimuth". Performance score relative to "Solar Altitude" based on distance The significance of the next largest cluster 54 measured is small. Therefore, this constitutes the labeling condition, enabling the computer-implemented method to group the visual parameters "Sun_Altitude" and "Sun_Azimuth" into the first cluster in the second visual parameter specification. Once labeled, validation can be performed based on the cluster subsets of the visual parameters.

[0067] Compared to Box 1, the given computer-implemented verification algorithm executed on the parameter specifications of Box 2 will run in a reduced amount of time. Therefore, parameters less relevant to the sensitivity of computer vision functions do not consume excessive computational resources during verification.

[0068] In an embodiment, providing a second visual dataset includes sampling at least one set of visual parameters using, for example, combined tests, search-based tests, or Latin hypercube sampling.

[0069] In an embodiment, the verification parameter specification includes at least one fewer visual parameter than the visual parameter specification, and / or at least one visual parameter of the verification parameter specification has a reduced numerical range compared to at least one visual parameter corresponding to the visual parameter specification, thereby providing a reduced verification parameter space.

[0070] In an embodiment, generating a verification visual parameter specification includes modifying at least one visual parameter set by dividing at least one visual parameter set of the visual parameter specification into at least first and second modified visual parameter sets, or by combining the first and second visual parameter sets of the visual parameter specification into a cluster.

[0071] In one embodiment, a first visual dataset is provided based on sampling of visual parameters according to a visual parameter specification, and a second visual dataset is provided based on sampling of visual parameters according to a validation parameter specification, wherein the sampling technique used for the visual parameters of the validation parameter specification is different from the sampling technique used for the visual parameters of the visual parameter specification.

[0072] Figure 3 The flowchart illustrating the verification function provided by the computer vision model is shown in the schematic diagram.

[0073] exist Figure 3 In, it has the same Figure 2 Functional blocks with the same functionality described herein are provided with the same numbering. Block 50 represents the verification function implemented by steps 110 and 112 of the method according to the first aspect. Optionally, the method outputs detailed test results 49. If the computer vision model 16 fails the verification process by repeatedly providing insufficient verification test results, it can be replaced with a new or redesigned computer vision model 16. Optionally, the computer vision model 16 can be tested as part of an autonomous system or robot provided as a "test tool".

[0074] Figure 4 An example of a computer vision model validation function is illustrated schematically.

[0075] The validation of computer vision model 16 will now be discussed. Based on the validation parameter specification obtained in step 110, function 50 tests the computer vision function 16 used to generate the validation parameter specification. A variety of existing computer vision function testing techniques can be applied to the computer vision model 16 represented by test engine 51.

[0076] For example, search-based tests, combinatorial tests, or statistical analyses using existing sampling techniques such as random or hypercube sampling can be applied in 51. The same procedure can be applied: Test Engine 51 generates a set of test parameters sampled from the validation parameter set.

[0077] For verification purposes, a second visual dataset is obtained by function 42 based on a set of test parameters supplemented by associated fundamental facts. Obtaining the second visual dataset based on the method used to obtain the input image provided as the first visual dataset in step 106 is not required.

[0078] Computer vision model 16 (which is considered the system under test) executes on a second vision dataset and associated fundamental facts. For each image, image set, or video in the second vision dataset, the output from computer vision model 16 is obtained. A comparison metric 48 between the output of computer vision model 16 and the fundamental facts is calculated and optionally fed back to test engine 51 to refine the generation of test cases (e.g., as applied in search-based testing). Test termination criteria are met, verification time has elapsed, or a given verification coverage has been achieved.

[0079] Performance scores available during and at the end of the validation process can optionally be used to provide validation test results 49. For example, on average, a weighted average or the worst validation result can be used from the comparison metric 48. Aggregation can be performed on all tests or for each test.

[0080] According to the embodiment, the verification test result 49 is output. For example, the verification test result 49 may be output as a machine-readable data structure or a machine-readable computer file, and / or displayed via a graphical user interface.

[0081] Therefore, the validation technique for computer vision model 16 was improved by using global sensitivity analysis and clustering. This improvement was achieved by selecting improved test parameters and value ranges.

[0082] The embodiments provide applying multiple validation tests to a computer vision model, wherein at least one validation test result includes an average, weighted average, or worst-case test result from the multiple test results.

[0083] Figure 5 An example of a global sensitivity analysis function for a computer vision model is illustrated schematically.

[0084] Generally speaking, sensitivity analysis (or more narrowly, global sensitivity analysis) can be viewed as a numerical quantification of how uncertainties in the output of a model or system can be partitioned and distributed to different sources of uncertainty in its inputs. This quantification can be called sensitivity or robustness. In the context of this specification, for example, a model can be considered as a mapping. From visual parameters (or visual parameter coordinates) based on which visual data items have been captured / generated / selected. To generate a performance score (or performance score coordinates) based on predictions and fundamental facts. If m equals one, there is only one performance score. However, m can be greater than one.

[0085] Variance-based sensitivity analysis (sometimes also called the Sobol method or Sobol index) is a specific type of (global) sensitivity analysis. For this purpose, the aforementioned mapping... Both the input and output samples can be interpreted probabilistically. In fact, as an example, an empirical (multivariate) distribution of the input samples can be generated. Similarly, for the output samples, an empirical (multivariate) distribution can be computed. The variance of the input and / or output (i.e., the performance score) can therefore be calculated. Variance-based sensitivity analysis can decompose the variance of the output into scores that can be attributed to the input coordinates or the set of input coordinates. For example, in the case of two visual parameters (i.e., n=2), one might find that 50% of the variance in the performance score is attributable to the first visual parameter (…). X 1) is caused by the variance, and 20% is caused by the second visual parameter ( X 2) is caused by the variance, and 30% is due to the interaction between the first and second visual parameters. For More than two visual parameters can interact. Note that if such interactions prove significant, combinations of two or more visual parameters can foster new visual dimensions and / or linguistic entities. Variance-based sensitivity analysis is an example of global sensitivity analysis.

[0086] Therefore, when applied in the context of this specification, a key outcome of variance-based sensitivity analysis is the variance of the performance score for each visual parameter. The larger the variance of the performance score for a given visual parameter, the greater the variation in the performance score of that visual parameter. This indicates that the computer vision model is less predictable based on the settings of that visual parameter. This unpredictability when training the computer vision model can be used to identify parameters that should be validated more carefully, and thus de-emphasize parameters that do not require validation at high resolution.

[0087] Figure 5 The example of sensitivity analysis function 19 operates on the performance score output from computer vision model 16, which is a combination of the original items of visual data (images) and their basis facts 15. The performance 17 of computer vision model 16 is evaluated for each pair of images and basis facts, thereby producing a maximum average performance 34 and a minimum average performance 35 for each pair of images and basis facts.

[0088] Optionally, for each image and fact pair provided using the same visual parameters 32, at least one of the maximum average performance 34 and the minimum average performance 35 is evaluated, or both of the maximum average performance 34 and the minimum average performance 35 are evaluated. This enables the computation of the performance variance 36 for each parameter (and sub-branch). The sensitivity analysis function 19 iterates over multiple visual parameters in the first visual parameter specification.

[0089] Iteration of the sensitivity analysis function 19 on each visual parameter in the first visual parameter specification is not required. Therefore, the performance variance 36 for each parameter (and optional subranges) in the first visual parameter specification is calculated.

[0090] The performance variance of 36 for each parameter enables performance rating of parameters (and sub-ranges) in rating unit 20. Optionally, multiple parameters can affect the performance of computer vision model 16, and these can be clustered into sub-regions.

[0091] Figure 6 An example illustrating how to transform visual parameter specification 70 into verification parameter specification 72 is shown.

[0092] For example, visual parameter specification 70 includes an optional visual parameter number field 74, a visual parameter name field 76, and a sensitivity score field 78. Following sensitivity analysis, the visual parameter specification summarizes the sensitivity of the computer vision model 16 to the input data. Validation parameter specification 72 is formed by selecting a subset of visual parameter specification 70 based at least on the sensitivity score field 78 for each visual parameter. For example, a validation threshold can be set, and if a visual parameter exceeds the validation threshold, it must be automatically included in the validation parameter specification.

[0093] Optionally, a verification resource metric can be provided based on the total amount of computation time or resources available for performing the verification process. The verification resource metric allows the visual parameters of visual parameter specification 70 to be included in verification parameter specification 72 based on the amount of computation time or resources available for performing the verification process.

[0094] As an example, a validation resource metric could be a vector defining multiple sensitivity categories for validation. For instance, any visual parameter with a computed sensitivity score above a first threshold (such as 80%) must be included in the validation parameter specification for validation at a first validation sampling rate. Similarly, any visual parameter with a computed sensitivity score below the first threshold but above a second threshold (such as 50%) is included in the validation parameter specification for validation at a second validation sampling rate different from the first. For example, the second validation sampling rate may be lower than the first validation sampling rate. For example, any visual parameter with a computed sensitivity score below the second threshold can be omitted from the validation parameter specification.

[0095] The embodiment provides: removing one or more visual parameters from the validation parameter specification, or adding one or more visual parameters to the validation parameter specification to form a modified validation parameter specification; providing a third visual dataset, which includes a third subset of visual data items conforming to the modified validation parameter specification and a corresponding third subset of basic fact data items; wherein the modified validation parameter specification is used to provide the third visual dataset; and applying additional validation tests to the computer vision model using the third visual dataset to obtain additional validation test results.

[0096] Figure 7 The illustration shows an example of how the verification coverage data structures 80 and 82 track the integrity of the verification process.

[0097] The first time point is illustrated in the validation coverage data structure 80, in which the visual parameter “precipitation” has been validated based on 95% sampling coverage, and “solar azimuth” has been sampled to 10% coverage.

[0098] For example, a predefined state for stopping verification can be calculated based on global sensitivity analysis. Figure 6 The "Solar Azimuth" parameter in the validation parameter specification appears to have a higher sensitivity score than the visual parameter "Precipitation". Therefore, the validation coverage for the "Solar Azimuth" visual parameter may need to be more complete than that for the "Precipitation" visual parameter. Therefore, in Figure 7 In the example, the predefined state used to stop the verification process is that the visual parameter "Sun_Azimuth" has received 95% verification coverage and the visual parameter "Precipitation" has received 60% verification coverage. Technicians will understand that many different predefined states can be provided to automatically stop the verification process based on verification coverage. The predefined state for stopping the verification process is proportional to, or calculated based on, the sensitivity of the visual parameters included in the verification parameter specification.

[0099] The embodiment provides that, for each validation test applied to a computer vision model, at least one visual parameter is stored in the validation coverage record, the at least one visual parameter describing each item of the visual data from a second vision dataset used to perform each validation test, and the validation test is stopped when the validation coverage record reaches a predefined state.

[0100] In an embodiment, performing sensitivity analysis further includes calculating multiple variances of multiple performance scores relative to the visual parameters of the visual parameter specification, and rating the visual parameters of the visual parameter specification based on the calculated multiple variances.

[0101] In one embodiment, at least one verification test result is displayed via a graphical user interface.

[0102] Figure 8 The apparatus according to the second aspect is illustrated schematically.

[0103] The second aspect provides an apparatus 300 for validating computer vision models, including an input interface 310, a processor 320, a memory 330, and an output interface 340.

[0104] Input interface 310 is configured to obtain a computer vision model, which is configured to perform computer vision functions representing elements of an observed scene, and to obtain a visual parameter specification including at least one set of visual parameters. The classification or regression performance of the computer vision model is affected by visual data items provided based on the at least one set of visual parameters. The input interface is configured to provide a first visual dataset, which includes a subset of visual data items conforming to the visual parameter specification, and a corresponding subset of basic fact data items.

[0105] Processor 320 is configured to perform sensitivity analysis on multiple performance scores output from a computer vision model over the domain of a visual parameter specification when multiple visual data items are applied to a visual dataset, to generate a validation parameter specification based on the visual parameter specification and the sensitivity analysis, wherein the validation parameter specification includes at least one fewer set of visual parameters or visual parameters compared to the visual parameter specification, and / or The validation parameter specification includes at least one set of visual parameters with a narrowed scope. The processor is further configured to validate the computer vision model by providing a second visual dataset comprising a second subset of visual data items conforming to the validation parameter specification, and a corresponding second subset of basic fact data items. The second visual dataset is provided using the validation parameter specification; and at least one validation test is applied to the computer vision model to obtain at least one validation test result.

[0106] In this example, the data processing device 300 is a personal computer, server, cloud-based server, or embedded computer. Processing does not necessarily occur on a single physical processor. For example, processing tasks can be partitioned across multiple processor cores on the same processor or across multiple different processors or virtual machines. The processor may be a Hadoop™ cluster or provided on a commercial cloud processing service. Some processing may be performed on unconventional processing hardware, such as a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), one or more graphics processors, a dedicated processor for machine learning, etc.

[0107] The memory 330 of device 300 stores a computer program 350 according to the fourth aspect, which, when executed by processor 320, causes processor 320 to perform the functions described by the computer-implemented methods according to the first and second aspects. According to the example, the input interface 310 and / or output interface 340 is a USB interface, an Ethernet interface, a WLAN interface, or one of other suitable hardware capable of enabling input and output of data samples from device 300. In the example, device 300 further includes a volatile and / or non-volatile memory system 330 configured to receive input observations as input data from input interface 310.

[0108] In this example, device 300 is an automotive embedded computer included in a vehicle, in which case the automotive embedded computer can connect to sensors and actuators present in the vehicle. For example, the input interface of the device can interface with one or more engine control units that provide speed, fuel consumption data, battery data, location data, etc. For example, the output interface 340 of device 300 can interface with one or more of a plurality of brake actuators, throttle actuators, fuel mixture or fuel-air mixture actuators, turbocharger controllers, battery management systems, automotive lighting systems, or entertainment systems, etc.

[0109] The third aspect provides a computer program including machine-readable instructions, which, when executed by a processor, is capable of performing (i) the computer-implemented method according to the first aspect, or (ii) the computer-implemented method according to the second aspect, or (iii) the computer-implemented method according to the third aspect.

[0110] The fourth aspect provides a computer-readable medium comprising at least one computer program according to the fifth aspect.

[0111] The examples provided in the accompanying drawings and those described in the foregoing written description are intended to provide an understanding of the principles of this specification. Therefore, no limitation is intended on the scope of the appended claims. This specification describes variations and modifications to the illustrated examples. Only preferred examples are presented, and all changes, modifications, and alternative applications of these examples within the scope of this specification are intended to be protected.

Claims

1. A computer-implemented method (100) for validating a computer vision model, comprising: - Obtain (102) a computer vision model, which is configured to perform computer vision functions that represent elements of the observed scene; - Obtain (104) a visual parameter specification comprising at least one set of visual parameters, wherein each set of visual parameters comprises one or more visual parameters, the one or more visual parameters comprising information about the content of the observed scene and / or representing boundary conditions for capturing and / or generating the observed scene, and wherein the classification or regression performance of the computer vision model is affected by visual data items provided based on the at least one set of visual parameters; - Provide (106) a first visual dataset, the first visual dataset comprising a subset of visual data items conforming to the visual parameter specification and a corresponding subset of basic fact data items; - When multiple visual data items are applied to a visual dataset, perform (108) sensitivity analysis on multiple performance scores output from the computer vision model over the domain of the visual parameter specification; - Generate (110) validation parameter specification based on visual parameter specification and sensitivity analysis, wherein the validation parameter specification includes at least one less set of visual parameters or visual parameters compared to the visual parameter specification, and / or wherein the validation parameter specification includes at least one set of visual parameters with a narrowed range; - Verify the (112) computer vision model through the following: - Provide (112a) a second visual dataset, the second visual dataset comprising a second subset of visual data items conforming to the validation parameter specification and a corresponding second subset of basic fact data items, wherein the validation parameter specification is used to provide the second visual dataset; and - Apply at least one validation test to the (112b) computer vision model to obtain at least one validation test result.

2. The computer-implemented method (100) according to claim 1, wherein, The multiple performance scores are provided through the following: - Use a computer vision model to generate multiple predictions of elements of a scene observed in a subset of visual data items, wherein the multiple predictions include at least one prediction of a classification label and / or at least one regression value of at least one item in the subset of visual data items; and - The multiple predictions of elements in a subset of visual data items are compared with corresponding subsets of basic fact data to obtain the multiple performance scores.

3. The computer-implemented method (100) according to claim 1 or 2. Multiple validation tests are applied to the computer vision model, wherein the result of at least one validation test includes the average, weighted average, or worst-case test result from the multiple test results.

4. The computer-implemented method (100) according to claim 1 or 2. The provision of a second visual dataset includes sampling the at least one set of visual parameters using one of a combined test, a search-based test, or Latin hypercube sampling.

5. The computer-implemented method (100) according to claim 1 or 2. The aforementioned multiple performance scores include or depend on any one or any combination of confusion matrix, precision, recall, F1 score, crossover ratio, or mean average.

6. The computer-implemented method (100) according to claim 1 or 2, further comprising: - Remove one or more visual parameters from the validation parameter specification, or add one or more visual parameters to the validation parameter specification to form a modified validation parameter specification; - Provide a third visual dataset, which includes a third subset of visual data items conforming to a modified validation parameter specification and a corresponding third subset of basic fact data items, wherein the modified validation parameter specification is used to provide the third visual dataset; - Apply additional validation tests to the computer vision model using a third vision dataset to obtain additional validation test results.

7. The computer-implemented method (100) according to claim 1 or 2. The first visual dataset is provided by sampling visual parameters based on the visual parameter specification, and the second visual dataset is provided by sampling visual parameters based on the validation parameter specification. The sampling technique for visual parameters applied to the validation parameter specification is different from the sampling technique for visual parameters applied to the visual parameter specification.

8. The computer-implemented method (100) according to claim 1 or 2, further comprising: - For each validation test applied to the computer vision model, at least one visual parameter is stored in the validation coverage record, the at least one visual parameter describing each item of the visual data from the second vision dataset used to perform each validation test; - Stop the verification test when the verification coverage records reach a predefined state.

9. The computer-implemented method (100) according to claim 1 or 2. The sensitivity analysis further includes: - Calculate the variances of the multiple performance scores relative to the visual parameters of the visual parameter specification; and - Rating visual parameters of the visual parameter specification based on multiple calculated variances.

10. The computer-implemented method (100) according to claim 1 or 2. The generation of the verification visual parameter specification includes modifying the at least one visual parameter set by dividing at least one visual parameter set of the visual parameter specification into at least first and second modified visual parameter sets, or by combining the first and second visual parameter sets of the visual parameter specification into a cluster.

11. The computer-implemented method (100) according to claim 1 or 2. in, Compared to the visual parameter specification, the validation parameter specification includes at least one fewer visual parameter, and / or wherein at least one visual parameter of the validation parameter specification has a reduced numerical range compared to at least one visual parameter corresponding to the visual parameter specification, thereby providing a reduced validation parameter space.

12. The computer-implemented method (100) according to claim 1 or 2, further comprising: - The results of at least one verification test are displayed via a graphical user interface.

13. An apparatus (300) for validating a computer vision model, comprising: - Input interface (310); - Processor (320); -Memory (330); and - Output interface (340); The input interface is configured to obtain a computer vision model, which is configured as follows: Perform computer vision functions to characterize elements of an observed scene and obtain a visual parameter specification comprising at least one set of visual parameters, wherein each set of visual parameters comprises one or more visual parameters, the one or more visual parameters including information about the content of the observed scene and / or representing boundary conditions for capturing and / or generating the observed scene, and wherein the classification or regression performance of the computer vision model is affected by visual data items provided based on the at least one set of visual parameters, and wherein the input interface is configured to provide a first visual dataset comprising a subset of visual data items conforming to the visual parameter specification and a corresponding subset of basic fact data items; The processor is configured to perform sensitivity analysis on multiple performance scores output from a computer vision model in the domain of a visual parameter specification when multiple visual data items are applied to a visual dataset, to generate a validation parameter specification based on the visual parameter specification and the sensitivity analysis, wherein the validation parameter specification includes at least one less set of visual parameters or visual parameters compared to the visual parameter specification, and / or wherein the validation parameter specification includes at least one set of visual parameters with a narrowed scope, and validates the computer vision model by: Provide a second visual dataset comprising a second subset of visual data items conforming to a validation parameter specification and a corresponding second subset of basic fact data items, wherein the second visual dataset is provided using the validation parameter specification; and apply at least one validation test to a computer vision model to obtain at least one validation test result.

14. A computer program comprising machine-readable instructions, which, when executed by a processor, is capable of implementing a computer-implemented method for verifying a computer vision model according to any one of claims 1 to 12.

15. A computer-readable medium comprising a computer program according to claim 14.

Images