Modifying a parameter set characterizing a computer vision model

Abstract

The present specification relates to a computer-implemented method for modifying a first visual parameter specification to provide a second visual parameter specification characterizing a computer vision model. Associated aspects relate to a computer-implemented method for providing a training data set, a computer-implemented method for training a computer vision model, an apparatus, a computer program and a computer readable medium. Computer vision focuses on how computers can automatically obtain a high-level understanding from digital images or videos. Computer vision systems are increasingly applied in the field of automotive or robotic vehicles. However, the validation of such systems is a complex task. Computer vision models used in computer vision systems are characterized in that a visual parameter specification defining an operational design region of the computer vision model is defined. The present specification discusses how to modify a visual parameter specification characterizing an operational design region of a computer vision model (16) by improving the visual parameter specification according to a sensitivity analysis of the computer vision model.

Description

Technical Field

[0001] This specification relates to a computer-implemented method for modifying a first vision parameter specification to provide a second vision parameter specification characterizing a computer vision model. Related aspects include computer-implemented methods for providing training datasets, computer-implemented methods for training computer vision models, apparatus, computer programs, and computer-readable media. Background Technology

[0002] Computer vision relates to how computers can automatically gain a high level of understanding from digital images or videos. Computer vision systems are increasingly being used in the automotive and robotic vehicle fields. Computer vision can process input from any interaction between at least one detector and its environment. The environment can be perceived by the at least one detector as a scene or a series of scenes. In particular, the interaction can be generated by at least one camera, a multi-camera system, a RADAR system, or a LIDAR system.

[0003] In automotive computer vision systems, while safety remains critical, computer vision often has to handle open environments. Therefore, it is important to consider quantitative safeguards when designing and testing computer vision models. Current computer vision development methods in autonomous driving heavily rely on expert (human) input, which reduces the predictability of computer vision models. Therefore, further improvements in computer vision are possible. Summary of the Invention

[0004] According to a first aspect, a method is provided for modifying a first visual parameter specification to provide a second visual parameter specification characterizing a computer vision model, wherein the method includes: - Obtain a computer vision model, which is configured to perform computer vision functions that represent elements of the observed scene; - Obtain a first visual parameter specification including at least one initial visual parameter set, wherein visual data items provided based on the range of the at least one visual parameter set can affect the classification or regression performance of the computer vision model; - Provide a visual dataset, which includes a subset of visual data items conforming to a first visual parameter specification and a corresponding subset of baseline truth data items; - Apply the subset of visual data items to the computer vision model to obtain multiple performance scores using corresponding benchmark ground truth data, the multiple performance scores characterizing the performance of the computer vision model when applied to a subset of visual data items in the visual dataset; - Perform sensitivity analysis of the plurality of performance scores over the domain of the first visual parameter specification; and - Generate a second visual parameter specification including at least one updated set of visual parameters, wherein the at least one initial set of visual parameters is modified based on the results of sensitivity analysis to provide at least one updated set of visual parameters.

[0005] The method according to the first aspect advantageously enables the automatic definition of a refined visual parameter specification that has been automatically improved to include one or more sets of visual parameters that are adjusted to avoid weaknesses of the target computer vision model (e.g., parameter combinations that lead to high variance).

[0006] Visual parameter specifications can be refined to include visual parameters that result in large output variances that contribute to performance scores. For example, images selected as training data according to a second visual parameter specification can be useful for training computer vision models because, by avoiding samples with high variance, images selected according to such a specification will not lead to training a computer vision model with input data that causes the model to perform poorly by providing incorrect predictions.

[0007] When training or validating a computer vision model, reducing one or more redundant parameters in the vision parameter specification (or the operational design domain of the computer vision model) is equivalent to a significant reduction in the size of the parameter space (and therefore, computational complexity), thereby enabling a faster and more energy-efficient training and / or validation process.

[0008] Visual parameters are any aspect of an image or video that affects a machine's vision capabilities in understanding a displayed scene. As an example, visual parameters could be the angle of the sun relative to the ego-vehicle, the time of day, the height and proximity of buildings and trees in the scene, the vehicle's speed, the speed and position of other vehicles relative to the vehicle, and the visual characteristics of the road surface in front of the vehicle.

[0009] In other words, the first step is to find the visual parameters that cause high variance in performance. However, the subsequent goal within these high-variance parameters is to partition / cluster such parameter values ​​(e.g., sun angle) into subranges with good performance and low variance (since the goal is to find the ODD), and to test this using a thorough validation step. Therefore, in one example, the ultimate goal is an ODD with low performance score variance.

[0010] As a concrete example, a wet road surface where the sun is directly in front of the vehicle can cause additional glare to be directed at the vehicle's windshield. Due to this increased glare, this condition can affect the accuracy with which a computer vision model classifies road signs located in front of the vehicle. Therefore, under the described conditions, the computer vision model is more likely to misidentify road signs in the road sign identification task. Consequently, the performance of the computer vision model exhibits a high variance in results.

[0011] During the training of a computer vision model, control over the training dataset enables the model to solve difficult situations. An Operational Design Domain (ODD) is defined, describing how computer vision features should be used. Typically, training a computer vision model should include the ODD; however, depending on the generalization performance of the CV features, they may perform well on the ODD even if not trained on it. Typically, the definition of the Operational Design Domain is a highly manual task requiring input from a domain expert.

[0012] According to this specification, a domain expert may provide an initial operational design domain, or the “first visual parameter specification” as described in this specification.

[0013] Once applied to computer vision models that provide input to the control systems of autonomous vehicles, robots, or drones, computer vision models trained according to this specification will perform more reliably, resulting in improved safety for autonomous vehicles, robots, or drones.

[0014] This specification describes a system that automatically analyzes the performance of a computer vision model on training data using an initial operational design domain and automatically provides a second operational design domain, or "second vision parameter specification," which has vision parameters that are improved through iterative refinement. The second vision parameter specification is provided based on a sensitivity analysis of the performance of the computer vision model on a test training set selected according to the initial vision parameter set.

[0015] In other words, if the initial set of visual parameters leads to poor performance of the computer vision model, such parameters can be reduced, narrowed, or removed in the second visual parameter specification. Specifically, the priority order of visual parameters can be determined, and a subrange of the initial visual parameters is defined as the safe operating region of the computer vision model. In some cases, visual parameters can be broken down into several sub-parameters.

[0016] The safe operating region is the area in which the performance of a computer vision model is relatively insensitive to changes in given visual parameters when identifying elements in the scene of the training data.

[0017] Generally, this method enables the automatic generation of operational design domains using a repeatable automated process, which allows for the resolution of ambiguous interpretations of visual parameters. Therefore, one or more visual parameter specifications generated according to the method of the first aspect can be used as test cases for empirical verification of a computer vision model implementing computer vision functions. The method implements decisions based on empirical results.

[0018] In other words, the first aspect analyzes which visual parameters have the greatest impact when testing or statistically evaluating a computer vision model. Given a set of visual parameters and a computer vision model as input, the output is a ranked list of visual parameters. By selecting a sublist of visual parameters from the ranked list, a simplified input model (ontology) for the second visual parameter specification is defined.

[0019] According to an embodiment, a computer-implemented method for modifying a first visual parameter specification to provide a second visual parameter specification, wherein the second visual parameter specification is used to provide training data for training a computer vision model.

[0020] The second aspect provides a computer-implemented method for providing training datasets, including: - Obtain the second visual parameter specification generated according to the first aspect or its embodiments; and - Obtain the training dataset, wherein the training dataset is obtained through one or a combination of the following steps: - Generate a synthetic training dataset using a synthetic vision data generator, the synthetic training dataset comprising synthetic vision data synthesized according to the second vision parameter specification and benchmark ground truth data; and / or - Sample visual data items from a database containing sample images associated with corresponding baseline truth data items, based on a second set of visual parameters; and / or - Specify the experimental requirements according to the second set of visual parameters, and perform field experiments to obtain the training dataset; and - Output the training dataset.

[0021] The advantage is that the generated training data is less likely to cause unpredictable performance of the computer vision model during training, thus resulting in an improved computer vision model.

[0022] The third aspect provides a computer implementation method for training computer vision models, including: - Obtain another computer vision model configured to perform computer vision functions representing elements of the observed scene; and - Obtain the training dataset generated according to the second aspect, and - Train the computer vision model using the training dataset.

[0023] Advantageously, it can provide computer vision models that are trained more predictably.

[0024] The fourth aspect provides an apparatus and a computer-implemented method for modifying a first visual parameter specification to provide a second visual parameter specification characterizing a computer vision model, the apparatus including an input interface, a processor, a memory, and an output interface.

[0025] The input interface is configured to obtain a computer vision model, which is configured to perform computer vision functions that characterize elements of the observed scene and obtain a first visual parameter specification including at least one initial visual parameter set, wherein generating visual data items based on the range of the at least one visual parameter set can affect the classification or regression performance of the computer vision model.

[0026] The processor is configured to provide a visual dataset conforming to a first visual parameter specification, wherein the visual dataset includes a subset of visual data items and a corresponding subset of benchmark ground truth data items, and is configured to apply the subset of visual data items to a computer vision model to obtain multiple performance scores characterizing the performance of the computer vision model against multiple visual data items and corresponding benchmark ground truth data, perform sensitivity analysis of the multiple performance scores on the domain of the first visual parameter specification, and generate a second visual parameter specification including at least one updated visual parameter set, wherein the at least one initial visual parameter set is modified based on the results of the sensitivity analysis.

[0027] The fifth aspect provides a computer program comprising machine-readable instructions that, when executed by a processor, are capable of performing (i) a computer-implemented method according to the first aspect, or (ii) a computer-implemented method according to the second aspect, or (iii) a computer-implemented method according to the third aspect.

[0028] The sixth aspect provides a computer-readable medium comprising at least one of the computer programs according to the fifth aspect.

[0029] Dependent embodiments of the foregoing aspects are given in the dependent claims and explained in the following description, which the reader should now refer to.

[0030] This definition has general applicability to this specification.

[0031] The visual dataset of the observed scene is a collection of items representing images or videos, such as JPEG or GIF images. Videos are sequences of images, optionally encoded in a format such as MPEG, if necessary.

[0032] A computer vision model is a function parameterized through model parameters. During training using machine learning techniques on a training dataset, these parameters are learned using, for example, machine learning techniques. The computer vision model is configured to map at least a visual data item, or a portion thereof, or a subset thereof, to predictions. One or more visual parameters define the visual state because they contain information about the content of the observed scene and / or represent the boundary conditions used to capture and / or generate the observed scene.

[0033] A ground truth data item corresponding to a visual data item is the classification and / or regression result that a computer vision model is designed to output in response to an image input when it is executed correctly. In other words, when given a visual data item as input that illustrates a predictable scene or scene elements, the ground truth data represents the correct or fairly accurate result of the computer vision model. The term "image" can refer to a subset of images, such as segmented road signs or obstacles.

[0034] The visual dataset of the observed scene is a collection of items representing images or videos, the latter being a sequence of images. Each visual data item can be a digital tensor with video, having additional dimensions for successive frames. A benchmark ground truth data item corresponding to a visual data item is, for example, the classification and / or regression result that a computer vision model should ideally output. For instance, if a visual data item is partially parameterized based on the presence of a wet pavement, and the presence or absence of a wet pavement is the expected output of the computer model to be trained, then the benchmark ground truth would return a description of that item in the associated visual data item as an image including the wet pavement.

[0035] Each ground truth data item can be another digital tensor associated with or referencing an item in the visual dataset, or, in a simpler case, a binary result vector. The ground truth data provides definitive results regarding scene elements in its associated visual data. For example, a visual data item containing an image of a 20 km / h speed limit sign would be associated with a result vector confirming that the scene contains a road sign, with a further level of detail: the road sign is a speed limit sign, and the displayed speed limit is 20 km / h. In this way, the correctness of the output predictions of the computer vision model applied to the visual data items can be evaluated by comparing them with the associated ground truth data, and a performance metric for the computer vision model can be calculated.

[0036] Exemplary embodiments of the invention are described in the accompanying drawings, which should not be construed as limiting the claims and are explained in more detail below. Attached Figure Description

[0037] Figure 1 The method implemented by a computer according to the first aspect is illustrated schematically.

[0038] Figure 2 An example of the automatic implementation of the first aspect of the first visual parameter specification is illustrated schematically.

[0039] Figure 3 An example of the sensitivity analysis function is illustrated schematically.

[0040] Figure 4 The set of visual parameters for clustering is illustrated schematically.

[0041] Figure 5a and 5b An example of subdividing parameter ranges is illustrated schematically.

[0042] Figure 6 The apparatus according to the fourth aspect is illustrated schematically. Detailed Implementation

[0043] One or more visual parameters define the visual state of a scene because they contain information about what is being observed in the scene and / or represent the boundary conditions used to capture and / or generate the observed scene. Typical scenes are characterized by a large number of visual parameters, some of which are continuous variables. Therefore, computing the search space of functions over the total number of visual parameters characterizing a typical scene is laborious.

[0044] Visual parameters can be, for example: camera properties (e.g., spatial and temporal sampling, distortion, aberrations, color depth, saturation, noise, etc.), LiDAR or RADAR properties (e.g., absorption), lighting conditions in the scene (light bounce, reflection, surface reflectivity, light sources, fog and light scattering, overall illumination, etc.), materials and textures, objects and their positioning, size, and rotation, geometry (of objects and the environment), parameters defining the environment, environmental characteristics (e.g., line-of-sight, precipitation characteristics, radiance (suspected to strongly interact with the detection process and may show a strong correlation with performance)), image characteristics / statistics (e.g., contrast, saturation, noise, etc.), domain-specific descriptions of the scene and situation (e.g., cars and objects at an intersection), etc. Many more parameters can be conceived.

[0045] These parameters can be viewed as ontology, classification, dimension, or linguistic entities. They can define a constrained view of the world or an input model. Given the assignment / selection of visual parameters, a specific set of images can be captured or rendered, or visual parameters can be used to describe images in an existing dataset. The advantage of using an ontology or input model is that, for testing, the expected test coverage target can be defined so that, for example, t-wise coverage can be used to define test termination criteria, and for statistical analysis, the distribution of these parameters can be defined.

[0046] Images, videos, and other visual data, along with other commonly annotated sensor data (GPS data, radiometric data, local meteorological characteristics), can be obtained in various ways. Real images or videos can be captured by image-capturing devices such as camera systems. Real images may already exist in a database, and a subset of images can be manually or automatically selected given visual parameters and / or other sensor data. Visual parameters and / or other sensor data can also be used to define the desired experiment. Another approach is to synthesize images given visual parameters and / or other sensor data. Images can be synthesized using image augmentation techniques, deep learning networks (e.g., generative adversarial networks (GANs), variational autoencoders (VAEs)), and 3D rendering techniques. Tools for 3D rendering in the context of driving simulations include, for example, the CARLA tool (Koltun, 2017, available at www.arXiv.org: 1711.03938).

[0047] Typically, in the development and testing of computer vision models, input images are defined, selected, or generated based on attributes (visual parameters) that appear important according to expert opinion. However, expert opinions regarding the correct selection of visual parameters may be incomplete or misled by assumptions arising from human perceptual experience. Human perception, based on the human perceptual system (the human eye and visual cortex), differs from the technical characteristics of detection and perception using computer vision models.

[0048] Conventionally, computer vision models may be developed, trained, or tested based on unrelated image properties, and visual parameters that are important influencing factors may be missed or underestimated. Furthermore, the technical system can detect additional characteristics such as polarization or extended spectral ranges imperceptible to the human perception system.

[0049] The computer vision model of an autonomous system is a safety-critical component. A computer vision model is configured to receive an input image or image sequence, apply computer vision functions to the input image or image sequence, and output a computer vision result. For example, the result could be a regression or recognition of elements within a scene depicted by the input image or image sequence. To ensure safety and for liability reasons, the definition of a vision parameter specification (sometimes called the operational design domain) is integrated with the computer vision model to provide safety assurance. Because the interaction between the computer vision model and its training data is complex, the operational design domain is defined.

[0050] Figure 1 The method implemented by a computer according to the first aspect is illustrated schematically.

[0051] The first aspect provides a method for modifying a first vision parameter specification to provide a second vision parameter specification suitable for providing data for training a computer vision model, wherein the method includes: - Obtain 102 computer vision models 16, which are configured to perform computer vision functions that represent elements of the observed scene; - Obtain a first visual parameter specification comprising 104 including at least one initial visual parameter set, wherein visual data items provided based on the range of the at least one visual parameter set are capable of influencing the classification or regression performance of the computer vision model; - Provide a 106-visual dataset, which includes a subset of visual data items conforming to the first visual parameter specification, and a corresponding subset of benchmark truth data items; - The subset of visual data items is applied to the computer vision model 108 to obtain multiple performance scores using corresponding benchmark ground truth data, the multiple performance scores representing the performance of the computer vision model when applied to a subset of visual data items of the visual dataset; - Perform sensitivity analysis of the multiple performance scores described in 110 on the domain of the first visual parameter specification; and - Generate 112 a second visual parameter specification including at least one updated set of visual parameters, wherein the at least one initial set of visual parameters is modified based on the results of sensitivity analysis to provide at least one updated set of visual parameters.

[0052] The computer vision model 16 is, for example, a deep neural network (DNN) comprising multiple neural network layers. However, other model topologies known to those skilled in the art can also be implemented according to this technique. The layers compute a latent representation, which is a higher-level representation of the input image or video sequence.

[0053] The first visual parameter specification is, for example, a data structure defining different types of visual phenomena that can affect the performance of a computer vision model performing functions such as image or video recognition, classification, or regression on a scene or part of a scene. Visual phenomena are encoded by one or more sets of visual parameters. For example, a visual parameter set can define parameters as a series of discrete values, such as precipitation intensity. Visual parameter sets can use Boolean conditions to define parameters, such as Boolean conditions defining the presence or absence of a tree cover. Alternatively or additionally, visual parameter sets can use continuous ranges (e.g., azimuth displacement) to define parameters. Sampling the set of values ​​within the boundaries defined by the visual parameter specification enables the provision of visual data items that conform to the visual parameter specification.

[0054] Visual parameter specifications can be thought of as representing ontology, classification, dimension, or linguistic entities that define a constrained view of the world or an input model. Specific image sets can be captured using visual parameters (from real-world image datasets) or rendered using a synthetic image generation engine.

[0055] In one embodiment, the visual parameters of the first or second visual parameter specification may include one or any combination of the following: one or more parameters describing the image capture arrangement, the configuration of optional image or video capture equipment, for optional spatial and / or temporal sampling, distortion aberrations, color depth, saturation, noise, absorption, one or more lighting conditions in the image / video scene, light bounce, surface reflectivity, reflection, light source, fog and light scattering, overall illumination, to obtain or synthesize visual data; and / or one or more features of the image / video scene, optionally, one or more objects and / or their positioning, size, rotation, geometry, material, texture, one or more parameters of the environment of the image / video capture equipment or the simulated capture equipment for the synthetic image generator, optionally, environmental characteristics, line-of-sight, precipitation characteristics, radiance, image characterization statistics such as contrast, saturation, noise, or one or more domain-specific descriptions of the image / video scene, optionally, one or more vehicle or road users, or one or more objects at an intersection.

[0056] In one embodiment, the domain of the first visual parameter specification includes a subset of numerical representations in a finite-dimensional vector space that allows the visual parameters to reside therein, optionally including a multidimensional interval of continuous or discrete visual parameters, or a set of numerical representations of visual parameters in a finite-dimensional vector space.

[0057] Box 1 is a practical example of a visual parameter specification. The text strings within each bracket (such as "spawn_point", "cam_yaw") are examples of initial visual parameter sets. In the example in Box 1, the values ​​of the initial visual parameter specification are a series of discrete values, although alternatively, at least one of the visual parameter specifications may include a range between at least two endpoints. Alternatively, at least one visual parameter set of the visual parameter specification may include multiple ranges (not illustrated in Box 1). Box 1 – An example of the first visual parameter specification in “Python”.

[0058] Images for training the computer vision model 16 are obtained based on a second visual parameter specification. To enable the computer vision model 16 to provide predictable representations of elements in a real-world scene, it should be trained on representative data. Typically, the visual parameter specification for box 1 is defined based on expert opinion. However, experts may have incorrect opinions or be biased. In such cases, the computer vision model may be trained on biased training data.

[0059] According to this specification, an updated or second visual parameter specification is returned after the optimization process. The second visual parameter specification is automatically modified according to the computer implementation method of the first aspect to ensure that, when visual data conforming to the definition of the second visual parameter specification is obtained, the visual data is not subject to undesirable constraints when applied to the computer vision model 16.

[0060] Many different modifications can be made to the visual parameter set. For example, at least one element of the visual parameter set can be deleted, added, reduced, or added. In Box 2, the upper limit range of the "cloudiness" and "precipitation" parameters is reduced from 100 to 85, while the range of the "sun altitude angle" parameter is narrowed by removing its last element. The "precipitation" parameter has been upsampled, and two additional elements have been added. The entire "cam yaw" parameter has been removed. These are concrete examples of how visual parameter specifications can be changed automatically, and many variations are possible. Box 2 – Example of the second visual parameter specification.

[0061] Figure 2 An example of the automatic implementation of the first aspect of the first visual parameter specification is illustrated schematically. A detailed description of an operational example based on the method of the first aspect is now provided.

[0062] As an example, an initial set of visual parameters and a set of values ​​or ranges of values ​​for the visual parameters can be defined (e.g., by an expert). A simple scene would have a first parameter that defines various solar altitudes relative to the vehicle's direction of travel, although a wider range of visual parameters is possible, as will be discussed later.

[0063] Sampling process 11 generates a set of value assignments for the visual parameter specification 10 (world model, ODD). Optionally, the parameter space defined by the visual parameter specification is randomly sampled according to a Gaussian distribution. Optionally, visual parameters are oversampled in regions where the performance inflection point of the computer vision model 16 is suspected. Optionally, visual parameters are undersampled in regions where the predictable performance of the computer vision model 16 is suspected. Therefore, in an embodiment, providing a visual dataset conforming to a first visual parameter set further includes sampling at least one initial visual parameter set included in the first visual parameter specification to obtain a sampled initial visual parameter value set, and using the sampled initial visual parameter value set to obtain the initial visual parameter values ​​in the visual dataset.

[0064] The next task is to acquire images according to visual parameter specifications.

[0065] A synthetic image generator, physical capture settings, and / or database selection 42 can be implemented to allow the generation, capture, or selection of images and corresponding reference truth terms based on visual parameters sampled during the sampling process 11.

[0066] For example, a synthetic image can be generated using the CARLA generator discussed earlier. In the case of synthetic generation, the baseline ground truth can be considered as sampled values ​​of the visual parameter specification used to generate a given synthetic image.

[0067] The physical capture settings enable experiments to be performed to obtain multiple test visual data sets within a specified parameter space. Alternatively, a database containing appropriately labeled archives of historical visual data can be selected.

[0068] In test step 17, the image from image acquisition step 42 is provided to computer vision model 16. Optionally, the computer vision model is included within the autonomous vehicle or robotic system 46 as an example of "hardware-in-the-loop" testing.

[0069] For each visual data item input into the computer vision model 16, a predicted output data item is obtained from the computer vision model 16. This represents a prediction of the elements of the scene represented in the visual data item input into the computer vision model 16. A performance score is calculated based on, for example, a comparison between a baseline ground truth and the predicted output data item. The result is multiple performance scores based on sampled values ​​according to the visual parameter specification.

[0070] Therefore, in one embodiment, obtaining multiple performance scores further includes: generating multiple predictions of elements of the observed scene in a subset of visual data items using a computer vision model, wherein the multiple predictions include at least one prediction of a classification label and / or at least one regression value of at least one item in the subset of visual data items; and comparing the multiple predictions of elements in the subset of visual data items with corresponding subsets of benchmark ground truth data to obtain multiple performance scores.

[0071] In one embodiment, the performance score includes or is based on any or a combination of the following: confusion matrix, precision score, recall score, F1 score, joint intersection score, or mean average score; and / or wherein the computer vision model is a neural network or a neural network-like model.

[0072] A global sensitivity analysis 19 was performed on the performance score for visual parameter 10. The global sensitivity analysis 19 determined the correlation between the visual parameters of the visual parameter specification and the performance of the computer vision model 16.

[0073] As an example, for each visual parameter, the variance of the performance score is determined. This variance is used to generate and / or display the ranking of the visual parameters. This information can be used to modify the initial set of 10 visual parameters in the visual parameter specification.

[0074] Therefore, in one embodiment, performing sensitivity analysis further includes: calculating multiple variances of each performance score among multiple performance scores relative to initial visual parameters of a first visual parameter specification; and ranking the initial visual parameters of the first visual parameter specification based on the calculated multiple variances.

[0075] Optionally, conditional clustering can be generated based on global sensitivity analysis 19. For example, if the first Boolean visual parameter "the camera is looking towards the sun" and the second Boolean visual parameter "the road is wet" are asserted, the performance of computer vision model 16 may be low, and both the first and second visual parameters may receive high rankings.

[0076] Step 17 and the global sensitivity analysis 19 can be repeated, and / or the computer vision model 16 can be retrained. Optionally, the performance score and its variance can be tracked during such training iterations. The training iterations are stopped when the variance of the performance score appears to have stabilized (stopped changing significantly). Optionally, irrelevant visual parameters can be iteratively modified, or irrelevant visual parameters can be removed from subsequent versions of the visual parameter specification. Optionally, higher resolution tests can be applied to portions of the visual parameter specification deemed important for the performance of feature 16.

[0077] Step 45 refines the visual parameter specification by using the results of global sensitivity analysis 19 to modify at least one initial visual parameter, thereby generating a second visual parameter specification. When the second visual parameter specification is sampled 11 and visual data 42 is obtained, multiple visual data items obtained according to the second visual parameter specification have improved performance scores.

[0078] As an example, a visual parameter with a performance score that has a low variance can be removed from the set of visual parameters. Alternatively, if the performance score improves above a threshold, another set of visual parameters can be added. The range of at least one visual parameter can be expanded or narrowed. Many different types of modifications can be performed on the first visual parameter specification.

[0079] Therefore, one embodiment further includes using the plurality of performance scores to identify at least one initial visual parameter set of a first visual parameter specification, and generating a second visual parameter specification includes modifying the at least one initial visual parameter set by expanding or shrinking its range over its domain, thereby producing a modified visual parameter set.

[0080] Typically, different sets of visual parameters (defining a world model or ontology) can be defined for testing or statistical evaluation of the computer vision model 16, and their implementations or precise interpretations can vary. This approach implements decisions based on empirical results 19, rather than solely on expert opinions, and its implementation is a concretization 42 of abstract parameters 10. Experts can still provide visual parameters as candidates 10.

[0081] Figure 2 The optional visual parameter specification verification step 47 is also illustrated. The second visual parameter specification is sampled based on the selection of the most relevant parameters and value ranges, for example, using combined testing, search-based testing, or statistical methods. Optionally, the test space is divided into value ranges and combinations, where the performance of the computer vision model 16 is categorized (e.g., good, bad, or variable). Variable categorization means that the performance of the computer vision model 16 is variable within a partition.

[0082] The visual parameter specification verification step 47 differs from the previous test performed via a combination of sampling process 11, visual data acquisition step 42, and testing step 17 because it allows for the use of alternative sampling algorithms and other metrics (such as test predictions), and the input space has a smaller dimension. Given a specific input space, this allows for a much higher test density. The goal of sensitivity analysis is to achieve this dimensionality reduction in the validation test space.

[0083] Therefore, in the embodiments, a process is provided to verify the second visual parameter specification 10b by the following steps: sampling the second visual parameter specification; providing another visual dataset including a subset of visual data items based on the second visual parameter specification that conforms to the second visual parameter specification and a corresponding subset of benchmark ground truth data items; and testing the subset of visual data items on the computer vision model 16 according to the dimensionality-reduced input space, thereby providing a verification result of the second visual parameter specification with reduced complexity.

[0084] In this way, a second visual parameter specification is provided as an optional output that can be used as a safety specification based on previously assumed norms. In subsequent training steps of another computer vision model, the second visual parameter specification can be sampled to provide visual data upon which the other computer vision model is trained. The ranking of visual parameters and their value ranges is another optional output.

[0085] Figure 3 An example of the sensitivity analysis function 19 is illustrated schematically.

[0086] Generally, sensitivity analysis (or global sensitivity analysis) can be viewed as a digital quantification of how the uncertainty in the output of a model or system can be partitioned and distributed to different sources of uncertainty in its input. This quantification can be referred to as sensitivity or robustness. In the context of this specification, the model can be viewed, for example, as a mapping from visual parameters, based on which visual data items have been captured / generated / selected to produce performance scores based on a comparison between baseline true values ​​and predictions. An important result of sensitivity analysis can be the variance of the performance score for each visual parameter. The larger the variance of the performance score for a given visual parameter, the greater the variation in the performance score for that visual parameter. This indicates that the computer vision model is less predictable based on the settings of that visual parameter.

[0087] In the context of this specification, the model can be considered, for example, from visual parameters (visual parameter coordinates). The mapping, Based on the aforementioned visual parameters (visual parameter coordinates) Visual data items have been captured / generated / selected to generate performance scores (or performance score coordinates) based on true and predicted baseline ground truth items (i.e., based on prediction and baseline ground truth). If m equals 1, there is only one performance score. However, m can be greater than 1.

[0088] Variance-based sensitivity analysis (sometimes called the Sobol method or Sobol index) is a specific type of (global) sensitivity analysis. For this purpose, samples of both the input and output of the aforementioned mapping can be interpreted probabilistically. In practice, as an example, a (multivariate) empirical distribution of the input samples can be generated. Similarly, for the output samples, a (multivariate) empirical distribution can be computed. The variances of the input and / or output (i.e., performance scores) can thus be calculated. Variance-based sensitivity analysis is able to decompose the variance of the output into fractions that can be attributed to the input coordinates or the set of input coordinates. For example, in the case of two visual parameters (i.e., n=2), one might find that 50% of the variance of the performance score is attributable to the first visual parameter (i.e., the first visual parameter). ) caused by the second visual parameter, 20% of the variance. The variance is caused by the interaction between the first and second visual parameters, with 30% of the variance attributable to this interaction. For n>2, more than two visual parameters interact. Note that if such interaction proves significant, a combination of two or more visual parameters can be enhanced to become a new visual dimension and / or linguistic entity. Variance-based sensitivity analysis is an example of global sensitivity analysis.

[0089] Therefore, when applied in the context of this specification, a key outcome of variance-based sensitivity analysis is the variance of the performance score for each visual parameter. The greater the variance of the performance score for a given visual parameter, the greater the variation in the performance score for that visual parameter. This indicates that the computer vision model is more unpredictable based on the settings of that visual parameter. Unpredictability during the training of the computer vision model 16 may be undesirable, and therefore, visual parameters that result in high variance can be weakened or removed during the training of the computer vision model.

[0090] In the context of this specification, the model can be viewed, for example, as a mapping from visual parameters upon which visual data items have been captured / generated / selected to produce performance scores based on true and predicted benchmark ground truth items. An important result of sensitivity analysis can be the variance of the performance score for each visual parameter. The larger the variance of the performance score for a given visual parameter, the greater the variation in the performance score for that visual parameter. This indicates that the computer vision model is more unpredictable based on the settings of that visual parameter.

[0091] Figure 3 An example of the sensitivity analysis function 19 operates on a performance score output from the computer vision model 16, combining raw visual data (images) items with their benchmark ground truth 15. The performance 17 of the computer vision model 16 is evaluated for each image and benchmark ground truth pair, thereby producing a maximum average performance 34 and a minimum average performance 35 for each image and benchmark ground truth pair.

[0092] Optionally, for each image and benchmark true value pair provided using the same visual parameters 32, at least one of the maximum average performance 34 and the minimum average performance 35 is evaluated, or both of the maximum average performance 34 and the minimum average performance 35 are evaluated. This makes it possible to calculate the performance difference for each parameter (and sub-branch) 36. Sensitivity analysis function 19 iterates over multiple visual parameters in the first visual parameter specification.

[0093] It is not necessary to iterate the sensitivity analysis function 19 on each visual parameter in the first visual parameter specification. Therefore, the performance variance 36 is calculated for each parameter (and optional subranges) in the first visual parameter specification.

[0094] The performance variance of 36 for each parameter enables performance ranking of parameters (and sub-ranges) in ranking unit 20. Optionally, multiple parameters can influence the performance of computer vision model 16, and these parameters can be clustered into sub-regions.

[0095] Figure 4 An example of a clustering visual parameter set is illustrated schematically. Figure 4 In Figure 50, the X-axis represents the sensitivity of the computer vision model 16's performance to given parameters. The Y-axis lists the visual parameters of the visual parameter specification shown in Box 1 above.

[0096] The visual parameters “sun_altitude” and “sun_azimuth” form the first visual parameter cluster 52. The visual parameters “cam_yaw”, “cam_pitch”, and “cam_roll” form the second visual parameter cluster 54. Because the sensitivity scores of each visual parameter in the first and second clusters satisfy the labeling criteria, the clusters are labelable.

[0097] The identifying criterion is the difference in magnitude between the two performance scores, "sun_altitude" and "sun_azimuth". According to distance The measured performance score for “sun_altitude” is relatively small compared to the significance of the next largest cluster, 54. Therefore, this constitutes an identification condition that enables the computer-implemented method to group the visual parameters “sun_altitude” and “sun_azimuth” into the first cluster in the second visual parameter specification.

[0098] Figure 5a and 5b An example of subdividing parameter ranges is illustrated schematically. Figure 5aThe diagram schematically illustrates multiple performance scores generated by the first visual parameter specification before segmentation. These scores can be evaluated... The ratio is used to evaluate the labeling condition. If this ratio is lower than the labeling threshold of a given subset of visual parameters, then that subset of visual parameters is a candidate for decomposition. Figure 5a As shown, the visual parameter subsets “sun_altitude” and “sun_azimuth” appear to have a much greater impact on the sensitivity of the performance of the computer vision model 16 than any other visual parameter.

[0099] Figure 5b An example of subdivided parameter ranges is illustrated schematically. In this case, "sun_altitude" and "sun_azimuth" have been automatically subdivided into four and three subranges, respectively. The metadata accompanying the visual parameters indicates that the domain of the "sun_altitude" visual parameter is between 0° and 90°. The metadata accompanying the visual parameters indicates that the domain of the "sun_azimuth" visual parameter is between 0° and 359°. Therefore, this method calculates the appropriate subdivision of the parameter range.

[0100] Optionally, the user can input suggestions for subdividing into sub-ranges via a graphical user interface. Optionally, the proposed subdivisions can be automatically presented to the user via the graphical user interface, and the proposed subdivisions are applied to the second visual parameter specification based on the user's consent.

[0101] In the example, once the subdivision of the parameter range is computed, a corpus of performance scores output from the computer vision model 16, combining the original terms of the visual data (images) and their previously computed baseline truth values ​​15, is resampled according to the subdivided visual parameter range. Thus, the subdivided visual parameter range returns more detail about the sensitivity of subranges of the visual parameter specification.

[0102] Optionally, the subdivided visual parameter ranges are incorporated into the second visual parameter specification.

[0103] Therefore, the embodiments further include identifying at least one initial visual parameter set of the first visual parameter specification using multiple performance scores based on identification conditions. Generating the second visual parameter specification includes modifying the at least one initial visual parameter set by dividing it into at least a first modified visual parameter set and a second modified visual parameter set, or combining the first and second visual parameter sets into a cluster.

[0104] A specific embodiment of the first aspect involves the step of sampling a first visual parameter specification.

[0105] Sampling of the at least one initial set of visual parameters is performed using a combination test or by Latin hypercube sampling. An example of an implementation of pairwise sampling functionality for combining visual parameters on visual parameters of a visual parameter specification is given in Box 3. Box 3 – Example Python code for implementing pairwise sampling functionality 11 of the first visual parameter specification.

[0106] Advantageously, this sampling technique enables good coverage of the entire parameter space with an efficient number of computations. In particular, it eliminates the need for brute-force sampling of the parameter space. Therefore, the technique described herein primarily enables the sampling of large parameter spaces using, for example, combined tests to determine the performance of sensitivity analysis.

[0107] The sensitivity analysis results in an initial reduction of search space complexity by providing a second visual parameter specification with visual parameters most relevant to the performance of the computer vision model. Then, in an optional validation step 47, the second visual parameter specification can be sampled again using techniques such as combinatorial testing to further reduce the search space and improve efficiency when validating the second visual parameter specification.

[0108] In one embodiment, a graphical representation of a second visual parameter specification is provided to a user via a graphical user interface displayed on an output interface, optionally combined with a graphical representation of a first visual parameter specification, and optionally combined with a graphical representation of sensitivity analysis results.

[0109] In one embodiment, a third visual parameter specification is generated as a result of interactive user modification of a graphical representation of the second visual parameter specification via a graphical user interface.

[0110] The fourth aspect provides an apparatus for testing the consistency between images generated by a synthetic image generator and images obtained from real visual data, the apparatus including an input interface, a processor, a memory, and an output interface.

[0111] The input interface is configured to obtain a computer vision model, which is configured to perform computer vision functions that characterize elements of the observed scene and obtain a first visual parameter specification including at least one initial visual parameter set, wherein generating visual data items based on the range of the at least one visual parameter set can affect the classification or regression performance of the computer vision model.

[0112] The processor is configured to: provide a visual dataset conforming to the first visual parameter specification, wherein the visual dataset includes a subset of visual data items and a corresponding subset of benchmark ground truth data items; apply the subset of visual data items to the computer vision model to obtain multiple performance scores characterizing the performance of the computer vision model for multiple visual data items and corresponding benchmark ground truth data, to perform a sensitivity analysis of the multiple performance scores in the domain of the first visual parameter specification; and generate a second visual parameter specification including at least one updated visual parameter set, wherein the at least one initial visual parameter set is modified based on the results of the sensitivity analysis.

[0113] Figure 6 The apparatus 300 according to the fourth aspect is schematically illustrated. In the example, the data processing apparatus 300 is a personal computer, a server, a cloud-based server, or an embedded computer. Processing does not necessarily occur on a single physical processor. For example, processing tasks can be partitioned across multiple processor cores on the same processor or across multiple different processors or virtual machines. The processor may be Hadoop™ clustering or provided on a commercial cloud processing service. Part of the processing may be executed on unconventional processing hardware such as field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), one or more graphics processors, dedicated processors for machine learning, etc.

[0114] The memory 330 of device 300 stores a computer program 350 according to the fourth aspect, which, when executed by processor 320, causes processor 320 to perform the functions described in the computer-implemented methods according to the first and second aspects. According to the example, the input interface 310 and / or output interface 340 is one of a USB interface, an Ethernet interface, a WLAN interface, or other suitable hardware that enables input and output of data samples from device 300. In the example, device 300 further includes a volatile and / or non-volatile memory system 330 configured to receive input observations as input data from input interface 310.

[0115] In this example, device 300 is an automotive embedded computer included in a vehicle, in which case the automotive embedded computer can be connected to sensors and actuators present in the vehicle. For example, the input interface of the device can be coupled to one or more of an engine control unit that provides speed, fuel consumption data, battery data, location data, etc. For example, the output interface 340 of device 300 can be coupled to one or more of a plurality of brake actuators, throttle actuators, fuel mixture or fuel-air mixture actuators, turbocharger controllers, battery management systems, automotive lighting systems, or entertainment systems, etc.

[0116] The fifth aspect provides a computer program comprising machine-readable instructions that, when executed by a processor, are capable of performing (i) a computer-implemented method according to the first aspect, or (ii) a computer-implemented method according to the second aspect, or (iii) a computer-implemented method according to the third aspect.

[0117] The sixth aspect provides a computer-readable medium including at least one of the computer programs according to the fifth aspect.

[0118] The examples provided in the accompanying drawings and described in the foregoing written description are intended to provide an understanding of the principles of this specification. Therefore, they are not intended to limit the scope of the appended claims. This specification describes variations and modifications to the illustrated examples. Only preferred examples are given, and it is intended to protect all changes, modifications, and further applications thereof within the scope of this specification.

Claims

1. A computer implementation of modifying a first visual parameter specification to provide a second visual parameter specification characterizing a computer vision model, wherein, Visual parameter specification is the operational design domain of a computer vision model, and the methods therein include: - Obtain (102) a computer vision model (16), which is configured to perform computer vision functions that represent elements of the observed scene; - Obtain (104) a first visual parameter specification comprising at least one initial visual parameter set, wherein visual data items provided based on the range of the at least one visual parameter set are capable of influencing the classification or regression performance of the computer vision model; - Provide (106) a visual dataset, the visual dataset including a subset of visual data items conforming to the first visual parameter specification, and a corresponding subset of benchmark truth data items; - Apply the subset of visual data items (108) to the computer vision model to obtain multiple performance scores, the multiple performance scores using corresponding benchmark ground truth data to characterize the performance of the computer vision model when applied to a subset of visual data items of the visual dataset; - Perform sensitivity analysis of the multiple performance scores described in (110) on the domain of the first visual parameter specification; and - Generate (112) a second visual parameter specification including at least one updated visual parameter set, wherein the at least one initial visual parameter set is modified based on the results of sensitivity analysis to provide at least one updated visual parameter set.

2. The computer-implemented method (100) according to claim 1, wherein obtaining multiple performance scores further comprises: - Generate multiple predictions of elements of an observed scene in a subset of visual data items using a computer vision model, wherein the multiple predictions include at least one prediction of the classification label and / or at least one regression value of at least one item in the subset of visual data. and - Multiple predictions of elements in a subset of visual data items are compared with corresponding subsets of benchmark ground truth data to obtain multiple performance scores.

3. The computer-implemented method (100) according to claim 1 or 2, further comprising: -Based on the identification conditions, multiple performance scores are used to identify at least one initial set of visual parameters for the first visual parameter specification, and The generation of the second visual parameter specification includes: modifying the at least one initial visual parameter set by dividing it into at least a first modified visual parameter set and a second modified visual parameter set, or combining the first visual parameter set and the second visual parameter set into a cluster.

4. The computer-implemented method (100) according to any one of the preceding claims. - Use multiple performance scores to identify at least one initial set of visual parameters for the first visual parameter specification, and in, Generating the second visual parameter specification involves modifying the at least one initial visual parameter set by scaling up or down its domain, thereby producing a modified visual parameter set.

5. The computer-implemented method (100) according to any one of the preceding claims, wherein, The sensitivity analysis further includes: - Calculate the variances of each performance score in a plurality of performance scores for the initial visual parameters with respect to the first visual parameter specification; and - Rank the initial visual parameters of the first visual parameter specification based on the calculated multiple variances.

6. The computer-implemented method (100) according to any one of the preceding claims, wherein, Providing a visual dataset that conforms to the first visual parameter specification further includes: - Sample at least one initial visual parameter set included in the first visual parameter set to obtain a sampled initial visual parameter value set; and - Obtain a visual dataset of initial visual parameter values ​​using the sampled initial visual parameter value set.

7. The computer-implemented method (100) of claim 6, wherein the sampling of the at least one initial visual parameter set is performed using a combination test or by Latin hypercube sampling.

8. The computer-implemented method (100) according to any one of the preceding claims, wherein, The domain of the first visual parameter specification includes a subset of numerical representations of visual parameters that are allowed to reside in a finite-dimensional vector space, optionally a multidimensional interval of continuous or discrete visual parameters, or a set of numerical representations of visual parameters in a finite-dimensional vector space.

9. The computer-implemented method (100) according to any one of the preceding claims verifies the second visual parameter specification through the following steps: - Sample the second visual parameter specification; - Provide another visual dataset including a subset of visual data items based on the second visual parameter specification that conforms to the second visual parameter specification and a corresponding subset of benchmark ground truth data items, and - The subset of visual data items is tested using a computer vision model (16) based on the reduced-dimensional input space to provide verification results of the second visual parameter specification with reduced complexity.

10. The computer-implemented method (100) according to any one of the preceding claims further comprises: - A graphical representation of the second visual parameter specification is displayed to the user via a graphical user interface on the output interface, optionally combined with a graphical representation of the first visual parameter specification, and optionally combined with a graphical representation of the results of the sensitivity analysis; and optionally: - As a result of interactive user modification of the graphical representation of the second visual parameter specification via the graphical user interface, a third visual parameter specification is generated.

11. A computer-implemented method for providing a training dataset, comprising: - Obtain a second visual parameter specification generated according to any one of the preceding claims; and - Obtain the training dataset, wherein the training dataset is obtained through one or a combination of the following steps: - A synthetic training dataset is generated using a synthetic vision data generator, the synthetic training dataset comprising synthetic vision data synthesized based on a second set of vision parameters and benchmark ground truth data; and / or - Sample visual data items from a database containing sample images associated with corresponding baseline truth data items, based on the second set of visual parameters; and / or - Specify the experimental requirements based on the second visual parameter set and perform on-site experiments to obtain the training dataset; and - Output the training dataset.

12. A computer-implemented method for training a computer vision model, comprising: - Obtain another computer vision model, which is configured to perform computer vision functions that characterize elements of the observed scene; and - Obtain the training dataset as generated according to claim 11; - Use the training dataset to train a computer vision model.

13. An apparatus (300) for modifying a first visual parameter specification to provide a second visual parameter specification characterizing a computer vision model, wherein, Visual parameter specifications are the operational design domain for computer vision models, including: - Input interface (310); - Processor (320); - Memory (330); and - Output interface (340); The input interface is configured to: obtain a computer vision model (106), the computer vision model (106) being configured to perform computer vision functions representing elements of the observed scene; and obtain a first visual parameter specification including at least one initial visual parameter set, wherein generating visual data items based on the range of the at least one visual parameter set can affect the classification or regression performance of the computer vision model; The processor is configured to: provide a visual dataset conforming to the first visual parameter specification, wherein the visual dataset includes a subset of visual data items and a corresponding subset of benchmark ground truth data items; apply the subset of visual data items to the computer vision model to obtain multiple performance scores characterizing the performance of the computer vision model for multiple visual data items and corresponding benchmark ground truth data, to perform a sensitivity analysis of the multiple performance scores in the domain of the first visual parameter specification; and generate a second visual parameter specification including at least one updated set of visual parameters, wherein the at least one initial set of visual parameters is modified based on the results of the sensitivity analysis.

14. A computer program comprising machine-readable instructions, which, when executed by a processor, are capable of performing (i) a computer-implemented method according to any one of claims 1 to 10, or (ii) a computer-implemented method according to claim 11, or (iii) a computer-implemented method according to claim 12.

15. A computer-readable medium comprising at least one of the computer programs of claim 14.

Images