Systems and methods for processing and using trained models

Abstract

The present invention relates to trained models trained on training data, such as trained neural networks. Systems and computer-implemented methods are provided for generating metadata encoding numerical characteristics of the training data of the trained model, and using the metadata to determine the consistency of input data of the trained model with the numerical characteristics of the training data. If the input data does not conform to the numerical characteristics, the trained model can be used for the input data to be considered non-compliant ("non-compliant"). Thus, a system that applies the trained model to the input data can, for example, warn a user of the non-compliance, or can refuse to apply the trained model to the input data, etc.

Description

Technical Field

[0001] The present invention relates to systems and computer-implemented methods for processing trained models, such as trained neural networks, which are trained on training data. The present invention also relates to systems and computer-implemented methods for using the trained models with input data, such as for classifying the input data. The present invention also relates to computer-readable media including instructions for performing any of the computer-implemented methods. Background Art

[0002] Machine learning is playing an increasingly important role in various fields and for various purposes. For example, in the field of medicine, machine learning techniques such as deep learning have been found to be well suited for classifying and segmenting image content from various modalities including CT, X-ray, digital pathology, and MRI. As is well known, such machine learning can use training data as input to train a model (e.g., a neural network). After such training, the trained model can be applied to new input data, for example, to obtain predictions or classifications from the new data. In a specific example, the trained model can be trained on labeled medical images and, during later use, applied to medical images to obtain classification or segmentation of anatomical structures in the medical images. Various other uses of the trained model are known.

[0003] Trained models are typically optimized and validated on a specific dataset. For example, in the example of medical image classification, the model may be trained on images and labels (or other classifications / annotations) from a specific modality. Users may expect the trained model performance to be always accurate. However, this may require that the input data (e.g., new images) conform to the same or similar characteristics as the training data. In practice, there is no guarantee that a trained model will be used under specified conditions (also known as "compliant" or "compliant" use), and it may be used "out of compliance."

[0004] For example, a user can apply a trained model to images acquired with a different image acquisition modality ("scanner") than that used to acquire the training images. Another example is that the scanner's settings may have changed, image quality may have degraded due to scratches on the lenses or imperfections in the scanning beam, the radiation dose may be different than required, etc. Summary of the invention

[0005] It may be desirable to facilitate compliant use of trained models.

[0006] The following aspects of the invention may involve generating metadata encoding numerical characteristics of training data for a trained model, and using the metadata to determine consistency of input data for the trained model with the numerical characteristics of the training data. Thus, a system that applies a trained model to input data may, for example, warn a user that use of the trained model with the input data is considered non-compliant (“non-compliant”), or may refuse to apply the trained model to the input data, etc.

[0007] According to a first aspect of the present invention, a system for processing a trained model is provided.

[0008] The system comprises:

[0009] Data interface, which is used to access:

[0010] model data, which represents the trained model, and

[0011] training data on which the trained model is trained;

[0012] A processor subsystem configured to:

[0013] The training data is characterized by the following operations:

[0014] applying the trained model to the training data to obtain an intermediate output of the trained model, and

[0015] determining the numerical characteristic based on the intermediate output of the trained model;

[0016] encoding the numerical property as metadata; and

[0017] The metadata is associated with the model data to enable an entity to apply the trained model to input data and thereby obtain a further intermediate output of the trained model to determine whether the input data conforms to the numerical characteristics of the training data of the trained model based on the further intermediate output.

[0018] According to another aspect of the present invention, there is provided a computer-implemented method for processing a trained model, comprising:

[0019] access:

[0020] model data, which represents the trained model, and

[0021] training data on which the trained model is trained;

[0022] The training data is characterized by the following operations:

[0023] applying the trained model to the training data to obtain an intermediate output of the trained model, and

[0024] determining the numerical characteristic based on the intermediate output of the trained model;

[0025] encoding the numerical property as metadata; and

[0026] The metadata is associated with the model data to enable an entity to apply the trained model to input data and thereby obtain a further intermediate output of the trained model to determine whether the input data conforms to the numerical characteristics of the training data of the trained model based on the further intermediate output.

[0027] According to another aspect of the present invention, a system for using a trained model is provided. The system comprises:

[0028] Data interface, which is used to access:

[0029] model data, which represents a trained model that has been trained on the training data,

[0030] metadata associated with the model data and comprising a numerical property, wherein the numerical property is determined from an intermediate output of the trained model when the trained model is applied to the training data, and

[0031] input data to which the trained model is to be applied;

[0032] A processor subsystem configured to:

[0033] applying the trained model to the input data to obtain a further intermediate output of the trained model,

[0034] determining, based on the further intermediate output, whether the input data conforms to the numerical characteristics of the training data of the trained model; and

[0035] If it is determined that the input data does not conform to the numerical characteristic, an output signal is generated indicative of the non-conformance.

[0036] According to another aspect of the present invention, there is provided a computer-implemented method for using a trained model, comprising:

[0037] access:

[0038] model data, which represents a trained model that has been trained on the training data,

[0039] metadata associated with the model data and comprising a numerical property, wherein the numerical property is determined from an intermediate output of the trained model when the trained model is applied to the training data, and

[0040] input data to which the trained model is to be applied;

[0041] applying the trained model to the input data to obtain a further intermediate output of the trained model,

[0042] determining, based on the further intermediate output, whether the input data conforms to the numerical characteristics of the training data of the trained model; and

[0043] If it is determined that the input data does not conform to the numerical characteristic, an output signal is generated indicative of the non-conformance.

[0044] According to another aspect of the present invention, there is provided a computer-readable medium including transient or non-transient data representing instructions arranged to cause a processor system to perform any one or both of the computer-implemented methods.

[0045] The above measures include accessing a trained model, such as a trained neural network, and accessing training data on which the trained model was trained. The trained model can be applied to the training data in a manner such that the training data is input data to be processed by the trained model. For example, if the trained model is trained for image classification, the training data can be used as input to the trained model to perform image classification on the training data.

[0046] The above measures also involve determining numerical characteristics based on intermediate outputs of the trained model. Here, the term "intermediate output" is to be understood as: the trained model processes input data to generate output data, such as classification, probability, etc. In order to generate the output data, the trained model generates intermediate output data, which are internal precursors of the output data. As a non-limiting example, in the case of a trained neural network, when the internal data is applied to the input data, such internal data can be the activation values ​​of the hidden units of the trained neural network. Other types of intermediate outputs are also conceivable and depend on the type of trained model. For example, if the trained model is a trained capsule network, the intermediate output can be one or more activity vectors of the capsule. Similar examples also exist for other models that can be trained by machine learning, such as latent variables in graphical models such as Bayesian networks.

[0047] The intermediate outputs may effectively represent the trained model's internal and intermediate representation of the input data, and when the input data is training data, an intermediate representation of the training data. By obtaining the numerical properties of the intermediate outputs, the intermediate properties may be made tangible. Thus, the numerical properties of the training data are obtained, albeit in an indirect rather than direct manner. Therefore, numerical properties are referred to above and below as "of" the training data, even though they are not calculated directly from the training data but indirectly from the intermediate outputs.

[0048] The inventors have considered that such numerical properties of the intermediate outputs of the trained model may be more suitable for characterizing the training data than the training data itself. That is, the training data may have properties that make it less suitable for such characterization. For example, the training data may be large in terms of data size, which may make it computationally complex to calculate descriptive numerical properties, and / or it may be difficult to manually (heuristically) or otherwise determine concise features of the training data. On the other hand, the trained model is often trained to obtain concise features, such as classifications, probabilities, etc. However, while the output data itself (e.g., classifications or probabilities) is often not fully descriptive within the range of possible input data, the intermediate outputs are well suited to provide such characterizations.

[0049] The numerical properties may be encoded as metadata, and the metadata may be associated with the trained model, or more specifically, with the model data of the trained model, in any suitable manner, e.g., by including the numerical properties in the model data itself (e.g., as a file header, XML element, etc., or providing the metadata as a separate file, or in any other manner).

[0050] On the application side, for example, when a trained model is used on non-training input data, the metadata can be used to determine whether the current input data of the trained model conforms to the characteristics of the training data, thereby determining whether to use the trained model, indicating "compliant" use or "non-compliant" use of the current input data. That is, the intermediate output of the trained model can be compared with the numerical characteristics encoded in the metadata or verified in another way to determine compliance or non-compliance. If non-compliance is detected, it can be used, for example, to warn the user, avoid classification through the trained model output, etc.

[0051] Optionally, the trained model is a trained neural network and the intermediate output comprises activation values ​​for a subset of hidden units of the trained neural network. The activation values ​​of the hidden units may be well suited for determining numerical properties describing the training data. A selection of such activation values ​​may be used. In some examples, the trained neural network may be a trained deep neural network, e.g., having several layers of hidden units. The selection of activation values ​​may correspond to a selection of a selected portion of a selected layer of the neural network.

[0052] Optionally, the training data comprises a plurality of training data objects, and the processor subsystem is configured to:

[0053] applying the trained model to individual objects in the plurality of training data objects to obtain a plurality of sets of activation values; and

[0054] The numerical characteristic of the training data is determined as a probability distribution over the plurality of sets of activation values.

[0055] The training data may include multiple individual training data objects, e.g., multiple images, audio recordings, etc. Typically, such training data objects may be characterized by a probability distribution. However, for many data types, it is either infeasible or at least computationally complex to extract the probability distribution directly from the training data itself, because the training data may be very large (e.g., many high-resolution images), so the probability distribution may also be too large and complex to be easily characterized, etc. Instead, for example, a probability distribution of activation values ​​selected by a hidden unit may be generated from the numerical outputs of a selected portion of a selected portion of a selected layer of a deep neural network. This may be computationally less complex because the number of hidden units may be much less than the number of data elements in each training data object.

[0056] Optionally, the processor subsystem is configured to:

[0057] obtaining non-compliant data, the non-compliant data comprising a plurality of non-compliant data objects having characteristics different from and inconsistent with the characteristics of the plurality of training data objects;

[0058] applying the trained neural network to individual objects in the plurality of non-compliant data objects to obtain another plurality of sets of activation values;

[0059] And selecting a subset of hidden units to establish, increase or maximize a difference between a) the probability distribution of the plurality of sets of activation values ​​and b) the probability distribution of the other plurality of sets of activation values.

[0060] It may be desirable to select a particular subset of the hidden units that produces a noticeable difference in the probability distribution when the probability distribution is calculated from the training data or the non-compliant data. Such non-compliant data, which when used as input to the trained model represents a non-compliant use of the trained model, can be considered, for example, by manual evaluation or by automatic metrics, because the super-non-compliant data does not conform to the characteristics of the training data, or at least does not conform to a sufficient degree. In particular, the subset can be selected so that a clear distinction is obtained between what is considered to be qualified data and non-compliant data. To this end, non-compliant data can be obtained, which includes a plurality of non-compliant data objects that are different from a plurality of training data objects. For example, in the case of images, the non-compliant data may include a set of images that have not been used in the training of the trained model and are considered to represent non-compliant input data for the trained model. The subset of hidden units from which the probability distribution is obtained can be selected to maximize the difference between the training data and the non-compliant data, or with, for example, a random selection or a selection of all hidden units.

[0061] For example, the processor subsystem may be configured to select the subset of hidden units by a combinatorial optimization method that optimizes the difference between a) the probability distribution of the plurality of sets of activation values ​​and b) the probability distribution of the other plurality of sets of activation values ​​as depending on the selected hidden units. The difference may be, for example, a Kullback-Leibler divergence measure, a cross entropy measure, or a mutual information measure known per se.

[0062] Optionally, the processor subsystem is configured to:

[0063] Using a generator portion of a generative adversarial network to generate negative samples based on the training data;

[0064] The non-compliant data is generated according to the negative sample.

[0065] The generator portion of a generative adversarial network (GAN) can be used to create non-compliant data in an automated manner, e.g., without or in conjunction with manual selection or manual generation of non-compliant data.

[0066] Optionally, the processor subsystem is configured to generate the model data by training the model using the training data, thereby obtaining the trained model. In some embodiments, the system that generates the metadata can be the same system as the system used to train the trained model. This may be convenient because the system already has access to the training data.

[0067] Optionally, the training data includes a plurality of images, and the trained model is configured for image classification or image segmentation.

[0068] Optionally, the system for using the trained model further comprises an output interface for outputting the output signal to a presentation device for presenting the output signal to a user in a sensory perceptible manner. For example, the system may generate a warning message on a display.

[0069] Optionally, the trained model is a trained neural network, the numerical characteristic is a probability distribution obtained from a plurality of sets of activation values ​​for a subset of hidden units of the trained neural network, the plurality of sets of activation values ​​being obtained by applying the trained model to the training data, another intermediate output of the trained model comprising the another set of activation values ​​for the subset of hidden units, and the processor subsystem of the system for using the trained model is configured to:

[0070] determining a probability of the another set of activation values ​​based on the probability distribution; and

[0071] A determination is made as to whether the input data conforms to the numerical characteristics of the training data of the trained model based on the probability.

[0072] The above may represent a specific example of how a system applying a trained model may determine whether input data conforms to the numerical characteristics of the training data of the trained model.

[0073] It will be appreciated by those skilled in the art that two or more of the above-mentioned embodiments, implementations, and / or optional aspects of the invention may be combined in any way deemed useful.

[0074] Based on this description, those skilled in the art can perform various modifications and variations of any computer-implemented method and / or any computer program product, which correspond to modifications and variations of the corresponding systems described. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] These and other aspects of the invention will become more apparent from and will be elucidated with reference to the embodiments described by way of example in the following description and to the accompanying drawings, in which:

[0076] Figure 1 A system for processing a trained model to generate metadata for the trained model, the metadata encoding numerical characteristics of the training data on which the trained model was trained, is shown;

[0077] Figure 2 A system for using a trained model with input data and using metadata to determine whether the input data conforms to numerical characteristics of the training data of the trained model is shown;

[0078] Figure 3 shows a detailed example of how to determine a numerical property for a trained neural network, in this example a probability function of the activation values ​​of a subset of the hidden units of the network;

[0079] Figure 4 Methods of processing a trained model to generate metadata encoding numerical characteristics of the model's training data are shown;

[0080] Figure 5 Methods are shown for using a trained model with input data and using metadata to determine whether the input data conforms to numerical characteristics of the training data for the trained model; and

[0081] Figure 6 A computer readable medium including data is shown.

[0082] It is noted that the drawings are purely diagrammatic and not drawn to scale. In the drawings, corresponding elements already described may have the same reference numerals.

[0083] List of abbreviations

[0084] DL Deep Learning

[0085] DNN Deep Neural Network

[0086] GAN Generative Adversarial Network

[0087] GenAl Genetic Algorithm

[0088] KL Kullback-Leibler

[0089] LIST OF REFERENCE NUMBERS The following list of reference numerals is provided to facilitate explanation of the drawings and should not be construed as limiting the claims.

[0090] 020, 022 Data storage devices

[0091] 030 Training Data

[0092] 040 Input data

[0093] 050 Model Data

[0094] 060 Metadata

[0095] 100 Systems for processing trained models

[0096] 120 data interface

[0097] 122, 124 data communication

[0098] 140 processor subsystem

[0099] 200 Systems using trained models

[0100] 220 data interface

[0101] 222, 224 data communication

[0102] 240 processor subsystem

[0103] 242 Output signal

[0104] 260 Display output interface

[0105] 262 Display data

[0106] 280 Display

[0107] 282 Warning dialog box

[0108] 300 training data

[0109] 310 Generative Adversarial Network

[0110] 320 trained models

[0111] 330 Probability Distribution (Compliant)

[0112] 335 Probability Distribution (Non-compliant)

[0113] 340 Difference measure (Kullback-Leibler divergence)

[0114] 350 Combinatorial Optimization Methods (Genetic Algorithms)

[0115] Activation values ​​and variances of 360 hidden unit subsets

[0116] 400 Methods for processing trained models

[0117] 410 Access model data, training data

[0118] 420 Characterization training data

[0119] 430 Applying the trained model to the training data

[0120] 440 Determine numerical characteristics

[0121] 450 Encoding Numerical Features as Metadata

[0122] 460Associating metadata with model data

[0123] 500 Ways to use trained models

[0124] 510 Access model data, metadata, input data

[0125] 520 Apply the trained model to the input data

[0126] 530 Determine the conformity of input data

[0127] 540 generates an output signal indicating non-compliance

[0128] 600 Computer readable medium

[0129] 610 Non-transient data DETAILED DESCRIPTION

[0130] Figure 1 A system 100 is shown for processing a trained model to generate metadata for the trained model, the metadata encoding numerical characteristics of the training data on which the trained model was trained. The system 100 may include a data interface 120 and a processor subsystem 140, which may communicate internally via data communication 124. The processor subsystem 140 may be configured to access model data 050 representing the trained model and access training data 030 on which the trained model was trained during operation of the system 100 and using the data interface 120. For example, Figure 1 As shown in , the data interface 120 can provide access 122 to an external data storage device 020 which may include the data 030, 050. Alternatively, the data 030, 050 can be accessed from an internal data storage that is part of the system 100. Alternatively, the data 030, 050 can be received from another entity via a network. In general, the data interface 120 can take various forms, such as a network interface to a local area network or a wide area network (such as the Internet), a storage interface to an internal or external data storage, etc. The data storage device 020 can take any known and suitable form.

[0131] The processor subsystem 140 may also be configured to characterize the training data during operation of the system 100 by applying the trained model to the training data to obtain an intermediate output of the trained model, and determine the numerical characteristic based on the intermediate output of the trained model. The processor subsystem 140 may encode the numerical characteristic as metadata 060 and associate the metadata 060 with the model data 050 to enable an entity to apply the trained model to the input data to determine whether the input data conforms to the numerical characteristic of the training data. An example of such an entity is Figure 2 system 200.

[0132] Metadata 060 may be stored by system 100 in data storage 020 or elsewhere, sent over a network, etc. Typically, metadata 060 may be stored in the same data container as training data 050, such as in the same file(s), but may also be provided as separate metadata 060 associated with training data 050. For example, in some embodiments, training data 050 may be linked to metadata 060, such as by including a URL at which metadata 060 may be accessed, or metadata 060 may be linked to training data 050. Various other contact methods are also contemplated and are within the capabilities of the skilled person.

[0133] Will refer to Figure 3 Various details and aspects of the operation of system 100, including optional aspects thereof, are further explained.

[0134] Typically, the system 100 can be implemented as a single device or device in or within a single device or apparatus, such as a workstation (e.g., a workstation based on a portable computer or a desktop computer) or a server. The device or apparatus may include one or more microprocessors that execute appropriate software. For example, the processor subsystem may be embodied by a single central processing unit (CPU), but may also be embodied by a combination or system of such CPUs and / or other types of processing units. The software may have been downloaded and / or stored in a corresponding memory, such as a volatile memory such as RAM or a non-volatile memory such as flash memory. Alternatively, the functional units of the system (e.g., a data interface and a processor subsystem) may be implemented in a device or apparatus in the form of a programmable logic device (e.g., a field programmable gate array (FPGA)). In general, each functional unit of the system may be implemented in the form of a circuit. Note that the system 100 may also be implemented in a distributed manner, such as in the form of cloud computing, involving different devices or apparatuses, such as distributed servers.

[0135] Figure 2 A system 200 is shown for using a trained model with input data and using metadata to determine whether the input data conforms to the numerical characteristics of the training data of the trained model. The system 200 may include a data interface 220 and a processor subsystem 240, which may communicate internally via data communication 224. The processor subsystem 240 may be configured to, during operation of the system 200 and using the data interface 220, as described with reference to Figure 1 The model data 050 and metadata 060 are accessed as described, as well as the input data 040 to which the trained model will be applied. Figure 2As shown in , the data interface 220 can provide access 222 to an external data storage device 022 including the data 040-060. Alternatively, the data 040-060 can be accessed from an internal data storage. Alternatively, the data 040-060 can be received via a network. In general, the data interface 220 can take various forms, such as a network interface to a local area network or a wide area network (such as the Internet), a storage interface to an internal or external data storage, etc. The data storage device 022 can take any known and suitable form.

[0136] The processor subsystem 240 may also be configured to, during operation of the system 200, apply the trained model 050 to the input data 040 to obtain another intermediate output of the trained model, determine whether the input data 040 conforms to the numerical characteristics of the training data of the trained model based on the further intermediate output, and if it is determined that the input data does not conform to the numerical characteristics, generate an output signal 242 indicating the non-conformance. As an optional component, the system 200 may include a display output interface 260 or any other type of output interface for outputting the output signal 242 to a presentation device, such as a display 280. For example, the display output interface 260 may generate display data 262 for the display 280, which causes the display 280 to present the output signal in a sensory perceptible manner, for example, as an on-screen warning dialog box 282.

[0137] Typically, the system 200 can be implemented as a single device or device in or within a single device or apparatus, such as a workstation (e.g., a workstation based on a portable computer or desktop) or a mobile device. The device or apparatus may include one or more microprocessors that execute appropriate software. For example, the processor subsystem may be embodied by a single central processing unit (CPU), but may also be embodied by a combination or system of such CPUs and / or other types of processing units. The software may have been downloaded and / or stored in a corresponding memory, such as a volatile memory such as RAM or a non-volatile memory such as flash memory. Alternatively, the functional units of the system (e.g., a data interface and a processor subsystem) may be implemented in a device or apparatus in the form of a programmable logic device (e.g., a field programmable gate array (FPGA)). In general, each functional unit of the system may be implemented in the form of a circuit. Note that the system 200 may also be implemented in a distributed manner, for example, involving different devices or apparatuses, such as a client and a server of a client-server implementation.

[0138] Figure 3 It is shown that, for example, Figure 1A detailed but non-limiting example of how the system 100 determines numerical properties for a trained model is provided. In this example, the trained model is a trained neural network, more specifically a deep neural network (DNN) model trained by deep learning (DL). For this purpose, deep learning techniques known per se can be used, which can be based on error function minimization, gradient descent methods, error back propagation and (particularly in deep learning) mini-batch optimization, etc. The aforementioned techniques can be considered textbook methods, but other suitable techniques can also be used. The trained model 320 can take the form of a deep neural network M with a suitable network architecture and a set of hidden units u. Figure 3 It is further shown that training data T can be used for multiple purposes. First, the training data T can be used to train a deep learning model M. Second, the training data T can be used to estimate a generative adversarial network (GAN) 310 (Goodfellow, 2014, see reference section), which can be used to generate so-called "non-compliant" samples that are close to the distribution of T that approximates the target, but are not actually derived from that distribution, for example, not actual observations like the training data T. Alternatively, the non-compliant data can be obtained from another source, such as from actual observations, such as from collected images that are believed to represent non-compliant images. The trained model M can be applied to both the non-compliant data and the training data T. This may result in the generation of model outputs, including all intermediate model outputs U.

[0139] Two probability distributions, namely P(U|Noncompliant) 335 and P(U|T) 330, can describe the sampling probability of the selected intermediate model unit output {u}. Such selected intermediate model units can, for example, correspond to selected hidden units of the trained model M, whose outputs correspond to their activation values. Such hidden units can also be simply called units, and their activation values ​​are called unit activations, which can also be simply referred to as intermediate model outputs. The normal distribution of the activation values ​​{u} of these units can be about the mean μ i and covariance σ ij where indices i and j correspond to all units in {u} that have been selected. Note that {u} can be a subset of the full set of hidden units in the model M. In some cases, a large number of units in {u} may require a simple distribution where, if i ≠ j, then σ ij = 0. However, in general, σ ij can be chosen to have or not have off-diagonal nonzero values. The choice of cells in the distribution and which σ ij Is non-zero can be Figure 3 In the expression {u}, {σ ij}360.

[0140] Given a choice {u}, {σij}, training data T and (GAN-generated) non-compliant data, the distributions P(U|non-compliant) and P(U|T) can be estimated, for example, by simply applying the trained model M to the data objects of the training data T and non-compliant data, respectively. ij The following choice of} may be preferred, where the difference between the two distributions is maximal, since the probability distribution is expected to be able to distinguish between non-compliant data from the training data T samples, or from the same source as the training data, from the above types and other types. This difference can be expressed by the Kullback-Leibler distance 340, cross entropy, mutual information or other measures known per se.

[0141] To maximize the difference between the distributions P(U|Noncompliant) and P(U|T), a combinatorial optimization method 350 may be used, e.g. from the family of genetic algorithms ('GenAl'), integer linear programming, etc. Alternatively, other combinatorial optimization methods known per se in the art may be used.

[0142] As a result, P(U|T) and P(U|Non-compliant) can be obtained, which can be optimal, or at least very suitable, to distinguish samples in the training data T from non-compliant data samples. P(U|Non-compliant) can be considered to be used to estimate whether a new input data object is considered non-compliant. However, it is preferred to use P(U|T) as a compliance detector instead, because one may aim for the training data T to be a complete description of the compliant data, while for non-compliant data, it may be difficult, if not impossible, to determine whether all non-compliant cases are covered. In this way, P(U|T) can be encoded as metadata of the deep learning model M. When using the deep learning model M on new input data, one can estimate whether the input data object (e.g., an image) is considered compliant or non-compliant by: using the input data object as input to the deep learning model M to estimate U new， The distribution function P(U|T) provided by the metadata is used to assign the qualified probability P(U new |T). For example, if the compliance probability P(U new |T) exceeds a preset threshold, the input data object can be considered compliant, otherwise it is non-compliant.

[0143] It is to be noted that when applying the trained model to images that are training data\non-compliant data or "new" input data, each image, when input to the trained model, may produce a vector in which each element I may correspond to the activation value of hidden unit i. This vector can be viewed as a sample drawn from an n-dimensional distribution (n is the number of hidden units considered / length of the vector), which can be described, for example, by a mean μ (of length n) and a variance matrix σ (nxn). During training, a plurality of such images may be distributed, for example, by a mean μ and a variance σ, which may be computed directly from the set of activation vectors. The above may also be applied to other training data, if necessary, including a plurality of training data objects.

[0144] Further reference to non-compliant data: In lieu of or in addition to generating this data by the GAN 310, the data may be collected from elsewhere, for example, in the same or similar manner as the training data T is collected from a medical scanning device. However, while the training data T may be annotated to perform supervised training of the model M, the non-compliant data may not be annotated as the model M is not trained on this data. For example, if the trained model is used for medical image classification, the non-compliant data may be obtained by filtering the recorded information of the medical scanning device for known non-compliant uses and using the images collected by such non-compliant uses as non-compliant data. For example, machine operation logs of CT scanners or pathology image scanners may be filtered, as well as possible recorded protocols, such as those representing the preparation of pathological tissue samples. Such excess data may also be collected in a similar manner in non-image-based medical environments, such as in the case of EEG or ECG data, or in non-medical environments in general. Thus, a non-compliant data object may be generated that helps to obtain P(U|T) that more accurately describes the non-compliant use, for example, to draw a clear boundary between compliant data and non-compliant data. Continue to refer to Figure 3 , the non-compliant data can be used as input / replacement of GAN at the deep learning model M.

[0145] In general, GAN 310 can be used to generate non-compliant data in a more general way because it does not rely on having to collect non-compliant data. Using a GAN generator directly may not be the best choice because the quality of the generator may be so high that it may be difficult for humans to distinguish the generated data from real data. However, it is known to apply GANs to generate so-called "negative samples", which can be used as non-compliant samples in this article. For example, this is known from Yu 2017, Dai 2017, Wang 2018, and Zheng 2018, which are incorporated herein by reference for describing the generation of such negative samples. Generating such negative samples may involve a trade-off between being outside of "compliant" characteristics (e.g., characteristics of the training data T) but close enough to these characteristics to allow definition of a clear boundary between non-compliant data and non-compliant data. Finally, a mixture of non-compliant data generated by GANs and non-compliant data obtained in other ways can also be used.

[0146] Although the numerical properties are described above as probability distributions, various other types of numerical properties may also be used, as known from the statistical field itself. Furthermore, if a probability distribution is used as a numerical representation, any known and suitable type, representation, or way of calculating a probability distribution may be used.

[0147] Figure 4 A block diagram of a computer-implemented method 400 for processing a trained model is shown. The method 400 may correspond to Figure 1 However, this is not limiting, as method 400 may also be performed using another system, apparatus, or device.

[0148] The method 400 may include, in an operation titled “Accessing Model Data, Training Data”, accessing 410 model data representing a trained model, and training data on which the trained model was trained. The method 400 may also include, in an operation titled “Characterizing Training Data”, characterizing 420 the training data by applying 430 the trained model to the training data in an operation titled “Applying Trained Model to Training Data” to obtain an intermediate output, and determining 440 numerical characteristics based on the intermediate output of the trained model in an operation titled “Determining Numerical Characteristics” of the trained model. The method 400 may also include, in an operation titled “Encoding Numerical Characteristics as Metadata”, encoding 450 the numerical characteristics as metadata, and in an operation titled “Associating Metadata with Model Data”, associating 460 the metadata with the model data to enable an entity that applies the trained model to input data to determine whether the input data conforms to the numerical characteristics of the training data of the trained model.

[0149] Figure 5A block diagram of a computer-implemented method 500 for using a trained model is shown. The method 500 may correspond to Figure 2 However, this is not limiting, as method 500 may also be performed using another system, apparatus, or device.

[0150] Method 500 may include, in an operation titled "Accessing model data, metadata, input data," accessing 510 model data representing a trained model that has been trained on training data, metadata associated with the model data, and including numerical properties, wherein the numerical properties are determined based on an intermediate output of the trained model and the input data to which the trained model is to be applied when the trained model is applied to the training data. Method 500 may also include, in an operation titled "Applying the trained model to the input data," applying the trained model to the input data 520 to obtain another intermediate output of the trained model. Method 500 may also include, in an operation titled "Determining compliance of input data," determining 530 whether the input data complies with the numerical properties of the training data of the trained model based on the further intermediate output. Such a determination of (non)compliance may, for example, include assigning a probability P(U new |T) is compared with the preset threshold θ, as previously described in Figure 3 Method 500 may further include, in an operation titled "GENERATING AN OUTPUT SIGNAL INDICTING A NON-CONFORMANCE", if it is determined that the input data does not conform to the numerical characteristic, generating 540 an output signal indicative of the non-conformance.

[0151] It should be understood that, generally, Figure 4 Method 400 and / or Figure 5 The operations of method 500 may be performed in any suitable order, such as serially, concurrently, or a combination thereof, depending upon where applicable, a specific order is required, such as by input / output relationships.

[0152] The method(s) may be implemented on a computer as a computer-implemented method, as dedicated hardware, or as a combination of both. Figure 6 Also shown, instructions for a computer, e.g., executable code, can be stored on a computer-readable medium 600, e.g., in the form of a series 610 of machine-readable physical indicia and / or as elements having different electrical, e.g., magnetic, or optical properties or values. The executable code can be stored in a transient or non-transient manner. Examples of computer-readable media include storage devices, optical storage devices, integrated circuits, servers, online software, etc. Figure 6An optical disc 600 is shown. Alternatively, the computer readable medium 600 may include transient or non-transient data 610 representing metadata as described elsewhere in this specification.

[0153] It will be appreciated that, although examples are given within the medical field, the techniques described in this specification may also be applied to a variety of other application areas, such as autonomous driving, where the input data may be sensor data and the trained model may be trained to make autonomous driving decisions based on the sensor data. Thus, for example, when the trained model is trained under summer driving conditions but is used under winter driving conditions, the user may be alerted if the sensor data does not conform to the characteristics of the sensor data on which the trained model was trained. In general, a trained model may be trained and used for classification or regression of input data.

[0154] For example, AI methods can be applied in traffic or crowd analysis, such as counting and tracking people and vehicles, and can be based on machine-learnable patterns trained on training data. Changing conditions, such as meteorological conditions (snow, fog, pollen) or operational conditions (extreme heat or cold), can affect the image and lead to poor AI performance. Irregular detections can alert human users to intervene.

[0155] In data traffic optimization, advanced machine learning algorithms can take large-scale, highly granular network data as input to generate precise demand forecasts for each node in the network and detect cross-temporal patterns in network traffic and utilization, which may also be based on machine learnable models trained on training data. Improved traffic and demand forecasts can enable more accurate assessments of network capacity requirements and reduce the need for over-provisioning of resources. However, the use of the network may vary over time or be completely disrupted (e.g., DDOS attacks). Early detection of such disruptions or changes in network traffic, in the form of non-compliance detection given a trained model, allows organizations to take proactive measures to ensure network performance.

[0156] References

[0157] Diederik P Kingma,MW(2013).Auto-Encoding Variational Bayes.Arxiv,arXiv:1312.6114.

[0158] Goodfellow,IJ(2014).Generative Adversarial Networks.arXiv,arXiv:1406.2661.

[0159] Yu,Y et al(2017).Open-Category Classification by Adversarial SampleGeneration.Proceedings of the Twenty-Sixth International Joint Conference onArtificial Intelligence(IJCAI-17)

[0160] Dai,Z.et al(2017).Good Semi-supervised Learning That Requires a BadGAN.31st Conference on Neural Information Processing Systems(NIPS2017),LongBeach,CA,USA.

[0161] Wang P.et al(2018).Incorporating GAN for Negative Sampling inKnowledge Representation Learning.AAAIPublications,Thirty-Second AAAIConference on Artificial Intelligence

[0162] Zheng,P.et al(2018).One-Class Adversarial Nets for FraudDetection.arXiv:1803.01798v2[cs.LG]5Jun 2018.

[0163] Examples, embodiments or optional features, whether or not indicated as non-limiting, should not be construed as limiting the invention as claimed.

[0164] It should be noted that the above-mentioned embodiment examples rather than limiting the present invention, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference numerals placed in brackets shall not constitute a limitation on the claims. The use of the verb "comprise" and its conjugated words does not exclude the presence of elements or steps other than those described in the claims. The word "one" or "an" in front of an element does not exclude the presence of multiple such elements. Expressions such as "at least one" before a list or element group represent the selection of all or any subset of elements from the list or group. For example, the expression "at least one of A, B, and C" should be understood to include only A, only B, only C, including both A and B, including both A and C, including both B and C, or or all of A, B, and C. The present invention can be implemented by means of hardware including several different elements and by means of appropriately programmed computers. In a device-type claim enumerating several devices, several of these devices can be implemented by the same hardware item. Although specific measures are recorded in mutually different dependent claims, this does not indicate that the combination of these measures cannot be used advantageously.

Claims

1. A system (100) for processing a trained model, comprising: A data interface (120) for accessing: Model Data (050), which represents the trained model, and training data (030), the trained model is trained on the training data, the training data comprising image data; A processor subsystem (140) configured to: The training data is characterized by the following operations: applying the trained model to the training data to obtain an intermediate output of the trained model, and determining a numerical characteristic based on the intermediate output of the trained model; encoding the numerical property as metadata (060); and associating the metadata with the model data to enable an entity to apply the trained model to input data and thereby obtain a further intermediate output of the trained model to determine whether the input data conforms to the numerical characteristic of the training data of the trained model based on the further intermediate output, wherein the trained model is a trained neural network, and wherein the intermediate outputs include activation values ​​for a subset of hidden units of the trained neural network.

2. The system (100) according to claim 1, wherein: The training data (030) includes a plurality of training data objects, and wherein the processor subsystem (140) is configured to: applying the trained model to individual training data objects in the plurality of training data objects to obtain a plurality of sets of activation values; and The numerical characteristic is determined as a probability distribution over the plurality of sets of activation values.

3. The system (100) according to claim 2, wherein: The processor subsystem (140) is configured to: obtaining non-compliant data, the non-compliant data comprising a plurality of non-compliant data objects having characteristics different from characteristics of the plurality of training data objects and not conforming to characteristics of the plurality of training data objects; applying the trained neural network to individual non-compliant data objects in the plurality of non-compliant data objects to obtain another plurality of sets of activation values; and The subset of hidden units is selected to establish, increase or maximize a difference between a) the probability distribution of the plurality of sets of activation values ​​and b) the probability distribution of the other plurality of sets of activation values.

4. The system (100) according to claim 3, wherein: The processor subsystem (140) is configured to select the subset of hidden units by a combinatorial optimization method, the combinatorial optimization method optimizing the difference between a) the probability distribution of the plurality of sets of activation values ​​and b) the probability distribution of the other plurality of sets of activation values ​​as dependent on the selected hidden units.

5. The system (100) according to claim 4, wherein: The processor subsystem (140) is configured to express the difference as or based on at least one of the following group: Kullback-Leibler divergence measure, The cross entropy measure, and Mutual information measure.

6. The system (100) according to any one of claims 3 to 5, wherein: The processor subsystem (140) is configured to: Using a generator portion of a generative adversarial network to generate negative samples based on the training data; The non-compliant data is generated based on the negative sample.

7. The system (100) according to any one of claims 1 to 5, wherein: The processor subsystem (140) is configured to generate the model data (050) by training a model using the training data (030), thereby obtaining the trained model.

8. The system (100) according to any one of claims 1 to 5, wherein: The training data (030) includes a plurality of images, and wherein the trained model is configured for image classification or image segmentation.

9. A computer-implemented method (400) of processing a trained model, comprising: access: model data, which represents the trained model, and training data on which the trained model is trained, the training data comprising image data; The training data is characterized by the following operations: applying the trained model to the training data to obtain an intermediate output of the trained model, and determining a numerical characteristic based on the intermediate output of the trained model; encoding the numerical property as metadata; and associating the metadata with the model data to enable an entity to apply the trained model to input data and thereby obtain a further intermediate output of the trained model to determine whether the input data conforms to the numerical characteristic of the training data of the trained model based on the further intermediate output, wherein the trained model is a trained neural network, and wherein the intermediate outputs include activation values ​​for a subset of hidden units of the trained neural network.

10. A system (200) for using a trained model, comprising: A data interface (220) for accessing: Model data (050) representing a trained model that has been trained on training data, the training data comprising image data, metadata (060) associated with the model data and comprising numerical properties, wherein the numerical properties are determined based on an intermediate output of the trained model when the trained model is applied to the training data, and Input data (040), the trained model is to be applied to the input data; A processor subsystem (240) configured to: applying the trained model to the input data to obtain a further intermediate output of the trained model, determining, based on the further intermediate output, whether the input data conforms to the numerical characteristics of the training data of the trained model; and If it is determined that the input data does not conform to the numerical characteristic, generating an output signal indicating non-conformity (242), wherein the trained model is a trained neural network, and wherein the intermediate outputs include activation values ​​for a subset of hidden units of the trained neural network.

11. The system (200) according to claim 10, further comprising an output interface (260) for outputting the output signal (242) to a presentation device (280) for presenting the output signal to a user in a sensory perceptible manner.

12. The system (200) according to claim 10 or 11, wherein: The numerical characteristic is a probability distribution obtained from a plurality of sets of activation values ​​of a subset of hidden units of the trained neural network, wherein the plurality of sets of activation values ​​are obtained by applying the trained model to the training data, and wherein another intermediate output of the trained model comprises another set of activation values ​​of the subset of hidden units, wherein the processor subsystem (240) is configured to: determining a probability of the another set of activation values ​​based on the probability distribution; and A determination is made as to whether the input data conforms to the numerical characteristics of the training data of the trained model based on the probability.

13. A computer-implemented method (500) of using a trained model, comprising: access: model data representing a trained model that has been trained on training data, the training data comprising image data, metadata associated with the model data and comprising a numerical property, wherein the numerical property is determined based on an intermediate output of the trained model when the trained model is applied to the training data, and input data to which the trained model is to be applied; applying the trained model to the input data to obtain a further intermediate output of the trained model, determining, based on the further intermediate output, whether the input data conforms to the numerical characteristics of the training data of the trained model; and If it is determined that the input data does not conform to the numerical characteristic, generating an output signal indicating non-conformity, wherein the trained model is a trained neural network, and wherein the intermediate outputs include activation values ​​for a subset of hidden units of the trained neural network.

14. A computer readable medium (600) comprising transient or non-transient data (610) representing instructions arranged to cause a processor system to perform a computer implemented method according to claim 9 or 13.

Images