Data-driven generalization analysis and improvement of deep learning models

Abstract

The present invention provides techniques for evaluating and limiting the scope of a data-driven deep learning model. In one embodiment, a machine-readable storage medium is provided, the machine-readable storage medium including executable instructions that, when executed by a processor, facilitate the execution of an operation, the operation including using a machine learning model to extract a first training data feature included in a training data set and a first target data feature included in a target data set. The operation also includes determining whether the target data set is within a limited data range of the training data set based on an analysis of the correspondence between the first training data feature and the first target data feature, and determining whether applying the target data set to a target neural network model developed using the training data set will generate a result with an acceptable level of accuracy based on whether the target data set is within the limited data range.

Description

Technical Field

[0001] The present application relates generally to deep learning, and more particularly to computer-implemented techniques for data-driven deep learning model generalization analysis and improvement. Summary of the invention

[0002] The following presents a summary of the invention to provide a basic understanding of one or more embodiments of the present invention. The present summary is not intended to identify key or important elements, nor is it intended to delineate any scope of a specific embodiment or any scope of the claims. Its sole purpose is to present concepts in a simplified form as a preface to a more detailed description presented later. In one or more embodiments described herein, a system, computer-implemented method, apparatus, and / or computer program product for data-driven deep learning model generalization analysis and improved techniques are provided.

[0003] According to one embodiment, a system may include a memory storing computer executable components and a processor executing the computer executable components stored in the memory. The computer executable components may include a data feature extraction component that uses a machine learning model to identify and extract training data features included in a training data set and target data features included in a target data set, wherein the target neural network model is trained on the training data set. For example, in some specific implementations, the machine learning model may include an adversarial autoencoder. The computer executable component may also include a data analysis component and a target data acceptability component, the data analysis component determines the degree of correspondence between the training data set and the target data set based on a comparison of the training data features and the target data features, and the target data acceptability component determines whether applying the target neural network model to the target data set will generate a result with an acceptable level of accuracy based on the degree of correspondence. In various specific implementations, the data analysis component may limit the scope of the training data set based on the training data features, and determine the degree of correspondence based on whether the target data features are within the scope of the training data set. The data analysis component may use a statistical analysis method or a machine learning method to determine the degree of correspondence.

[0004] In some embodiments, the target data acceptability component can determine whether applying the target neural network model to the target data set will generate results with an acceptable level of accuracy based on defining acceptability criteria. Using these embodiments, the target data acceptability component can determine that applying the target neural network model to the target data set will not generate results with an acceptable level of accuracy based on the degree of correspondence not satisfying the acceptability criteria.

[0005] In various embodiments, the training data features include first training data features, the target data features include first target data features, and the computer executable components further include a model application component that applies a subset of layers of the target neural network model to the training data set and the target data set based on determining that the degree of correspondence meets the acceptability criteria. Using these embodiments, the computer executable components may also include a model feature extraction component that extracts second training data features generated based on applying the subset of layers to the training data set, and extracts second target data features generated based on applying the subset of layers to the target data set. Also for these embodiments, the degree of correspondence determined by the data analysis component is a first degree of correspondence, and the computer executable components may also include a model analysis component that determines a second degree of correspondence between the second training data features and the second target data features. The target data acceptability component may also determine whether applying the target neural network model to the target data set will generate results with an acceptable level of accuracy based on the second degree of correspondence.

[0006] In some implementations of these embodiments, the model analysis component limits the scope of the target neural network model based on the second training data feature, and determines the second degree of correspondence based on whether the second target data feature is within the scope of the target neural network model. The model analysis component may also use a statistical analysis method or a machine learning method to determine the second degree of correspondence. For example, in some implementations, the data analysis component and the model analysis component may use the same statistical analysis method or the same machine learning method to determine the first degree of correspondence and the second degree of correspondence, respectively.

[0007] In one or more specific implementations, the target data evaluation component can determine that applying the target neural network model to the target data set will not generate results with an acceptable level of accuracy based on the second degree of correspondence not satisfying the acceptability criteria. Similarly, the target data evaluation component can determine that applying the target neural network model to the target data set will generate results with an acceptable level of accuracy based on a second determination that the second degree of correspondence satisfies the acceptability criteria for the second degree of correspondence.

[0008] In some embodiments, elements described in conjunction with the system can be embodied in a different form, such as a computer-implemented method, a computer program product, or another form. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A block diagram of an exemplary non-limiting system that facilitates evaluating and scoping a data-driven deep learning model in accordance with one or more embodiments of the presently disclosed subject matter is shown.

[0010] Figure 2Exemplary non-limiting feature extraction methods that facilitate evaluating and limiting the scope of a data-driven deep learning model in accordance with one or more embodiments of the presently disclosed subject matter are shown.

[0011] Figure 3 A flow chart illustrating an exemplary non-limiting process for evaluating and scoping input data for a data-driven deep learning model in accordance with one or more embodiments of the presently disclosed subject matter is shown.

[0012] Figure 4 A flow chart illustrating an exemplary non-limiting process for evaluating and scoping model-dependent features for a data-driven deep learning model in accordance with one or more embodiments of the presently disclosed subject matter is shown.

[0013] Figure 5 A block diagram of another exemplary non-limiting system that facilitates evaluating and scoping a data-driven deep learning model in accordance with one or more embodiments of the presently disclosed subject matter is provided.

[0014] Figure 6 A flow chart of an exemplary non-limiting computer-implemented method that facilitates evaluating and limiting the scope of a data-driven deep learning model in accordance with one or more embodiments of the presently disclosed subject matter is provided.

[0015] Figure 7 A flowchart is provided for another exemplary non-limiting computer-implemented method that facilitates evaluating and limiting the scope of a data-driven deep learning model in accordance with one or more embodiments of the presently disclosed subject matter.

[0016] Figure 8 A flowchart is provided for another exemplary non-limiting computer-implemented method that facilitates evaluating and scoping a data-driven deep learning model in accordance with one or more embodiments of the disclosed subject matter.

[0017] Fig. 9 A flowchart is provided for another exemplary non-limiting computer-implemented method that facilitates evaluating and limiting the scope of a data-driven deep learning model in accordance with one or more embodiments of the presently disclosed subject matter.

[0018] Fig.10 A flowchart is provided for another exemplary non-limiting computer-implemented method that facilitates evaluating and limiting the scope of a data-driven deep learning model in accordance with one or more embodiments of the presently disclosed subject matter.

[0019] Fig.11 A block diagram of an exemplary, non-limiting operating environment is shown in which one or more embodiments described herein may be facilitated. DETAILED DESCRIPTION

[0020] The following specific embodiments are merely exemplary and are not intended to limit the application or use of the embodiments and / or the embodiments. In addition, it is not intended to be bound by any express or implied information set forth in the aforementioned "background technology" or "content of the invention" section or "specific embodiments" section.

[0021] The present subject matter discloses systems, computer-implemented methods, devices and / or computer program products that help evaluate and limit the scope of data-driven deep learning models. A deep neural network (DNN) is an artificial neural network (ANN) with multiple layers between an input layer and an output layer. DNN finds the correct mathematical operation to transfer input to output, whether it is a linear relationship or a nonlinear relationship. DNN has recently shown impressive performance in various artificial intelligence (AI) fields (including computer vision, speech, natural language processing, bioinformatics, drug design, medical image analysis, etc.), sometimes exceeding humans. These achievements are made possible by the significant improvement in computing power and the availability of large-scale annotated data sets, resulting in better inference performance than traditional models. However, the data-driven nature of deep learning models leads to challenges during the inference stage. Specifically, one of the fundamental problems of data-driven machine learning methods is that the final model inference ability is limited by the scope of the training data used to develop the model.

[0022] The subject matter disclosed in the present invention relates to techniques for limiting the scope of application of a DNN model on new inference data based in part on the characteristics of the training data used to develop the model. Specifically, the techniques disclosed in the present invention use statistical-based methods and machine learning-based methods to evaluate the scope of application of the model due to the natural characteristics of the training data and the generalizability limitations of the final model. The techniques disclosed in the present invention also provide for identifying new data for potential processing by the model that may lead to inaccurate inferences, and determining confidence levels / confidence boundaries for ensuring that the model is accurately applied to the new data. In various embodiments, the techniques disclosed in the present invention use a two-step process to determine the scope of the DNN model and determine whether the DNN model can provide accurate inference results based on a specific new data set (referred to herein as a target data set). The first step is referred to herein as a data range evaluation step, and the second step is referred to herein as a model range evaluation step.

[0023] The first step of the two-step method, the data range assessment step, involves determining the range of the training data set (also referred to herein as reference data) used to develop and train the DNN model, and determining whether the target data set is within the range of the training data set. In this regard, the first step involves determining whether the new data set encountered by the DNN model in the field is contained in the statistical distribution of the training data set for training the DNN model. In one or more embodiments, the first step involves extracting a set of low-dimensional related data features from the training data set (or reference data set) used to train the DNN model. A set of low-dimensional features extracted from the training / reference data set is referred to herein as the first training data features. In various embodiments, supervised or semi-supervised machine learning feature extraction methods (such as using adversarial autoencoders, etc.) can be used to extract the first training data features from the training data set. The same feature extraction method can also be used to extract another set of low-dimensional related features (e.g., new data for potential processing by the DNN model) from the target data set. A set of low-dimensional features extracted from the target data set is referred to herein as the first target data features.

[0024] Statistical and / or machine learning analysis methods may then be used to limit the scope of the training / reference data set based on the first training data feature, and determine whether the target data set is within the scope of the training data set based on a corresponding comparison between the first training data feature and the first target data feature. For example, some suitable statistical analysis methods that can be used to determine whether the first target data feature is within the scope of the first reference data feature may include, but are not limited to: standard score (also known as Z score or Z value) analysis on each dimension, and Mahalanobis distance analysis on multiple dimensions. Some suitable machine learning analysis methods that can be used to determine whether the first target data feature is within the scope of the first reference data feature may include, but are not limited to: T-distributed stochastic neighbor embedding (t-SNE) analysis of lower dimensions to obtain similarity distances, one-class support vector machine (SVM) classification for anomaly detection, and isolation forests for anomaly detection.

[0025] If it is determined based on statistical and / or machine learning analysis that the first target data feature is outside the range of the first training data feature (e.g., meaning that a low degree of correspondence is observed between the first training data feature and the first target data feature), it can be assumed that the inference results generated based on applying the DNN model to the target data set will likely be inaccurate. Therefore, if the first target data feature is outside the range of the first training data feature, the target data set may be associated with a low confidence score, which represents a low confidence in the accuracy of the results generated based on applying the DNN model to the target data set. In some specific implementations, the confidence score may be associated with the level of predicted accuracy of the results generated based on applying the DNN model to the target data. For example, a confidence score of 80% may correspond to an accuracy level of 80%, etc. In some embodiments, the technology disclosed in the present invention may generate a notification or warning indicating that the target data set is determined to be associated with a low confidence score or otherwise outside the range of the training data set. The notification or warning can be used to prevent the DNN model from being applied to the target data set or otherwise adjust the downstream weighting of the inference generated by applying the DNN model to the target data set.

[0026] The second step of the two-step process, the model range assessment step, involves assessing the range of the DNN model based on applying the DNN model to the training / reference data set. In this regard, the structure of the DNN model, the number of hidden layers, the output parameters generated at each hidden layer, the number of parameters generated, etc. are controlled in part based on the training data used to develop and train the model. The second step of the two-step process is used to determine whether and with what confidence the DNN model can process the target data in a manner consistent with the training data set to produce accurate results based on the results generated based on the training data set.

[0027] In various embodiments, the second step in the two-step process is performed only when the target data set is determined to be within the range of the training data set according to the first step analysis in the two-step process. For example, in one or more embodiments, if the data evaluation process results in a determination that the target data set is associated with a confidence score / level that exceeds a minimum threshold, such that the target data set can be considered to be within the range of the training data set, then the technology disclosed in the present invention can further perform a model range evaluation. However, if the data evaluation process results in a determination that the target data set is associated with a low confidence score / level that does not exceed the minimum threshold, such that the target data set can be considered to be outside the range of the training data set, then the technology disclosed in the present invention can declare the target data set as not suitable for the DNN model and abandon the step of proceeding to the model range evaluation step.

[0028] According to the model scope assessment step, a subset of layers of the DNN model (e.g., one or more layers of the DNN model excluding the final output layer) may be applied to the training data set to generate a first set of model-based features based on the training data set. The first set of model-based features or a subset of the first set of model-based features may be extracted as second training data features. The same subset of layers of the DNN model may also be applied to the target data set to generate a second set of model-based features. The second set of model-based features or a subset of the second set of model-based features may be extracted as second target data features. Using the same statistical and / or machine learning analysis methods used in the first step, the scope of the DNN model may be further limited based on the second training data features, and the second target data features may be compared with the second training data features to determine whether the target data set is within the scope of the DNN model. If the second target data feature is outside the scope of the DNN model (e.g., if the second target data feature does not exhibit a high degree of correspondence with the second training data feature), the target data set may be associated with a low confidence that the DNN model will generate accurate results when applied to the target data set. A notification or warning may also be generated at this time, indicating that the target data set is determined to be associated with a low confidence score or otherwise outside the scope of the DNN model. This notification or warning may be used to prevent application of the DNN model to the target data or to otherwise adjust downstream weightings of inferences resulting from application of the DNN model to the target data.

[0029] However, if the second target feature is within the range of the DNN model (e.g., the second target data feature is within the range of the second reference data feature), the target data set can be associated with a high confidence that the DNN model will generate accurate results when applied to the target data set. In other words, if the second target feature is within the range of the second reference feature, the target data set can be associated with a high confidence that the model performance based on the target data set will have similar performance to the training / reference data set. In some embodiments, based on a first determination at a first step (data evaluation step) that the target data set is within the range of the reference data set and a second determination at a second step (model evaluation step) that the target data set is within the range of the DNN model, the target data can be authorized for application to the DNN model. In addition, the results generated based on applying the DNN model to the target data set can be associated with a high accuracy (e.g., according to a predefined accuracy scale).

[0030] One or more embodiments are now described with reference to the accompanying drawings, wherein the same reference numerals are used to represent the same elements throughout. In the following description, for the purpose of explanation, many specific details are set forth in order to provide a more thorough understanding of one or more embodiments. However, it is apparent that in various cases, one or more embodiments may be practiced without these specific details.

[0031] Now turning to the attached figure, Figure 1 A block diagram of an exemplary non-limiting system 100 is shown that facilitates evaluating and limiting the scope of a data-driven deep learning model in accordance with one or more embodiments of the subject matter disclosed herein. Embodiments of the system described herein may include one or more machine-executable components embodied in one or more machines (e.g., embodied in one or more computer-readable storage media associated with one or more machines). Such components, when executed by one or more machines (e.g., processors, computers, computing devices, virtual machines, etc.), may cause one or more machines to perform the operations described.

[0032] For example, in the illustrated embodiment, system 100 includes a computing device 102 that includes several machine-executable components, including (but not limited to) a data feature extraction component 104, a data analysis component 106, a target data acceptability component 108, a model application component 112, a model feature extraction component 114, a model analysis component 116, and a model acceptability component 118. System 100 also includes various data sources and / or data structures that include information that can be used and / or evaluated by one or more components of computing device 102. For example, in the illustrated embodiment, these data sources and / or data structures include a training data set 124, a target data set 126, and a target neural network model 128. Computing device 102 may also include or be operably coupled to at least one memory 120 and at least one processor 122. At least one memory 120 may also store executable instructions (e.g., data feature extraction component 104, data analysis component 106, target data acceptability component 108, model application component 112, model feature extraction component 114, and model analysis component 116), which, when executed by at least one processor 122, facilitates performing operations defined by the executable instructions. In some embodiments, the memory 120 may also store various data sources and / or structures of the system 100 (e.g., training data set 124, target data set 126, target neural network model 128, etc.). In other embodiments, the various data sources and structures of the system 100 may be stored in other memories (e.g., at a remote device or system) that the computing device 102 can access (e.g., via one or more networks). The computing device 102 may also include a device bus 110 that couples various components of the computing device to each other. Examples of the processor 122 and memory 120, as well as other suitable computer-based or computing elements, may be referenced. Fig.10 Find and combine Figure 1 Or use with one or more of the systems or components shown and described in other figures disclosed herein.

[0033] The system 100 and / or one or more components of the system 100 or other systems disclosed herein can be used to solve problems that are highly technical in nature, are not abstract, and cannot be performed by humans as a series of mental behaviors using hardware and / or software. In this regard, the system 100 and / or components of the system 100 can use deep learning, statistical analysis, and machine learning to determine and limit the scope boundaries of data-driven DNN models (e.g., target neural network models 128) developed and trained based on a large amount of training data (e.g., training data set 124). The system 100 and / or components of the system 100 also provide for determining data that causes model application failures and determining confidence regions / confidence boundaries for model application in the field. The system 100 and / or one or more components of the system 100 achieve these goals by using semi-supervised feature extraction methods to extract the most important low-dimensional features for the purpose of the DNN model (e.g., classification), and customize the application of the DNN model in the domain to improve generalization.

[0034] In this regard, some of the processes performed by the system 100 may be performed by a dedicated computer for performing defined tasks associated with evaluating a DNN model (e.g., the target neural network model 128) to automatically determine the scope of the DNN model after training and development of the DNN model. Based on the determined scope of the DNN model, one or more dedicated computers and / or components of the system 100 (as well as other systems described herein) may be used to automatically adjust the application of the DNN model to the new data set based on a determined confidence level that may be based on the accuracy of the results generated by the application of the DNN model to the new data set in the field. For example, the system 100 and / or one or more components of the system 100 may be used to prevent the application of the DNN model to a new data set that may produce inaccurate results, or otherwise facilitate adjustment of downstream weighting of inferences generated by applying the DNN model to a new data set that is determined to be outside the scope of the DNN model.

[0035] Thus, system 100 and / or components of system 100 (and other systems described herein) can help improve the accuracy of results generated based on the application of DNN models (e.g., target neural network model 128) in the field, while minimizing errors and downstream effects of inaccurate inferences generated based on the application of DNN models to new data. In this regard, system 100 and / or components of system 100 or other systems described herein can be used to solve new problems arising from advances in technology, computer networks, the Internet, etc., particularly advances in AI solutions based on data-driven DNN machine learning models.

[0036] In one or more embodiments, the system 100 can employ a two-step process to determine the scope of a particular DNN model (e.g., a target neural network model 128) that has been developed and trained on existing reference / training data, and to determine whether the DNN model can provide accurate inference results when applied to a new data set. In the illustrated embodiment, the particular DNN model to be evaluated is identified as the target neural network model 128, and the particular reference / training data set used to develop and train the target neural network model 128 is identified as the training data set 124. The new data set being evaluated for potential processing by the target neural network model 128 is identified as the target data set 126. The type of target neural network model 128 being evaluated can vary. For example, the target neural network model 128 may include a DNN model that provides automatic inference in associated computer vision, speech analysis, natural language processing, bioinformatics, drug design, medical image analysis, etc.

[0037] The first step of the two-step process (referred to herein as the data range assessment step) involves determining the range of the training data set 124 (also referred to herein as reference data) used to develop and train the target neural network model 128, and determining whether the target data set 126 is within the range of the training data set 124. In one or more embodiments, the data range assessment step involves the data feature extraction component 104, the data analysis component 106, and the target data acceptability component 108.

[0038] The data feature extraction component 104 can perform a feature extraction method based on machine learning to extract low-dimensional relevant features or feature vectors from the training data set 124 and the target data set 126. Specifically, the data feature extraction component 104 can use a suitable feature extraction method that automatically extracts a set of features or feature vectors for the two data sets, which reduces the dimensions of the corresponding data sets to a set of smaller representative feature vectors. The features or feature vectors extracted from the training data set 124 are referred to as first training data features or first training data feature vectors in this article. The feature vectors extracted from the target data set 126 are referred to as first target data features or first target data feature vectors in this article.

[0039] The data feature extraction component 104 may use the same feature extraction method to extract the first training data feature from the training data set 124 and the first target data feature from the target data set 126. In some embodiments, the data feature extraction component 104 may use an autoencoder model such as an adversarial autoencoder (AAE) model to extract the first training data feature and the first target data feature. For example, the AAE model may include a model trained on a large-scale data set (e.g., the training data set 124) with or without labeled or annotated training data.

[0040] Figure 2 An exemplary non-limiting feature extraction method 200 that facilitates evaluating and defining the scope of a data-driven deep learning model in accordance with one or more embodiments of the presently disclosed subject matter is shown. Figure 2 The AAE model is shown being applied to an input dataset (e.g., input dataset 202) to extract a set of low-dimensional correlated feature vectors (e.g., feature vector 206) from the input dataset.

[0041] exist Figure 1 Reference from the perspective of Figure 2 ,exist Figure 2 In the illustrated embodiment, the input data set 202 includes a set of raw medical images. For example, according to an exemplary implementation, the target neural network model 128 may include a medical image analysis model (e.g., a Pneumothorax detection model) that is configured to generate inferences (e.g., classification, diagnosis, etc.) based on analysis of a specific type of raw medical image. According to the technology disclosed in the present invention, the data feature extraction component 104 may use the data feature extraction method 200 to extract a set of feature vectors from the training data set 124 and another set of feature vectors from the target data set 126. In this regard, Figure 2 The input dataset 202 shown may correspond to an original medical image set (e.g., training dataset 124) used to develop and train the target neural network model 128, or a new original medical image set (e.g., target dataset 126) to be processed by the target neural network model 128.

[0042] According to the exemplary AAE feature extraction method 200, the feature vector 206 includes a set of low-dimensional feature vectors extracted from the input dataset 202. The feature vector 206 represents the input dataset 202 in a low-dimensional structured format, such that the feature vector 206 can be used to substantially automatically regenerate the initial high-dimensional raw input image with the minimum accuracy required for classification purposes. For example, the image dataset 204 provides an example automatically generated dataset that can be generated based on the feature vector 206. It can be seen that by comparing the image dataset 204 to the initial input dataset 202, the image generated based on the feature vector 206 substantially corresponds to the image of the input dataset 202. In this regard, the extracted (low-dimensional related) feature vector group 206 represents the original input image (e.g., the input dataset 202) in a unified manner having a reduced dimensionality relative to the original input image. Although Figure 2 The visualization illustrated in depicts a low dimensionality where two nodes Z are used to represent feature vectors from eight input images, but it should be understood that the dimensionality of the input dataset and the resulting extracted feature vectors may vary.

[0043] Reference again Figure 1, continuing with the data evaluation step, the data analysis component 106 may compare and analyze the first feature / feature vector extracted from the training data set 124 and the first feature / feature vector extracted from the target data set 126 to determine whether the target data set 126 is within the range of the training data set 124. In this regard, the data analysis component 106 may use one or more statistical and / or machine learning analysis methods to define the range of the training data set 124 based on the first training data feature / feature vector, and determine whether the target data set 126 is within the range of the training data set 124 based on the comparison of the correspondence between the first training data feature / feature vector and the first target data feature / feature vector. Some suitable statistical analysis methods that can be used to determine whether the first target data feature is within the range of the first reference data feature may include, but are not limited to: standard score (also known as Z score or Z value) analysis on each dimension, and Mahalanobis distance analysis on multiple dimensions. Some suitable machine learning analysis methods that can be used to determine whether the first target data feature is within the range of the first reference data feature may include, but are not limited to: t-SNE analysis of lower dimensions to obtain similarity distances, one-class SVM classification for anomaly detection, and isolation forests for anomaly detection.

[0044] In this regard, the data analysis component 106 may employ a suitable statistical and / or machine learning analysis method (e.g., those described above) to compare the first training data feature / feature vector with the first target data feature / feature vector to determine the degree of correspondence between the first training data feature / feature vector and the first target data feature / feature vector. Depending on the statistical / machine learning analysis method used, the degree of correspondence may be based on a Z-score, a distance measure, a distribution measure, a percentage value, a combination of two or more similarity measures, etc. In this regard, the specific type of measurement data used to represent the degree of correspondence between the first training data feature / feature vector and the first target feature / feature vector may vary based on the type of statistical and / or machine learning comparative analysis method employed.

[0045] The target data acceptability component 108 may also evaluate the correspondence degree measurement data determined by the data analysis component 106 to determine whether the target data set 126 exhibits an acceptable degree of correspondence with the training data set that is considered to be within the scope of the training data set 124. For example, in some embodiments, the target data acceptability component 108 may adopt predefined acceptability criteria for the correspondence degree measurement data. The predefined acceptability criteria may include, for example, predefined benchmark or threshold correspondence values. For example, the predefined acceptability criteria may include a minimum Z score, a maximum degree of deviation, a minimum corresponding percentage value, a maximum degree of separation, etc. According to these embodiments, the target data acceptability component 108 may determine whether the target data set 126 is within the scope of the training data set 124 or otherwise exhibits an acceptable degree of correspondence with the training data set 124 based on whether the correspondence degree measurement data meets the acceptability criteria. For example, in one embodiment, where the corresponding data includes a percentage value (e.g., X%) representing a degree of correspondence between a first target data feature / feature vector and a first training data feature / feature vector, and where the acceptability criteria includes a minimum percentage correspondence value (e.g., Y%), the target data acceptability component 108 can determine that the target data set 126 is within the range of the training data set 124 if the percentage of correspondence between the first target data feature / feature vector and the first training data feature / feature vector is greater than or equal to the minimum percentage of correspondence value (e.g., if X% ≥ Y%).

[0046] In various embodiments, if the target data acceptability component 108 determines that the first target data feature is outside the range of the first training data features (e.g., meaning that a low degree of correspondence is observed between the first training data feature and the first target data feature), it can be assumed that the inference results generated based on applying the target neural network model 128 to the target data set 126 may be inaccurate. Therefore, if the first target data feature is outside the range of the first training data features (e.g., meaning that the degree of correspondence between the first target data feature and the first training data feature does not meet the acceptability criteria), the target data acceptability component 108 can associate the target data set 126 with a low confidence score, which represents a low confidence in the accuracy of the results generated based on applying the DNN model to the target data set.

[0047] In some implementations, the confidence score may reflect a binary representation of a confidence level in the target data set 126. For example, the confidence score associated with the target data set 126 associated with the first step, i.e., the data analysis evaluation, may reflect a "yes" or a "no", where the "yes" indicates that the target data set is within the range of the training data set and therefore will generate results with an acceptable level of accuracy, and the "no" indicates that the target data set is outside the range of the training data set and therefore will generate results with an unacceptable level of accuracy. In other embodiments, the confidence score may be associated with a predicted confidence level of the accuracy of the results generated based on applying the DNN model to the target data. For example, a confidence score of 80% may correspond to a predicted confidence level, i.e., the results are accurate 80% of the time, and so on. According to these embodiments, the degree of correspondence between the first training data feature / feature and the target data feature / feature vector is directly related to the confidence score associated with the target data set 126, such that the higher the degree of correspondence, the higher the confidence score. The target data acceptability component 108 may also employ a predefined confidence level scoring scheme that associates a defined confidence score with a defined degree of correspondence.

[0048] In some embodiments, the degree of correspondence and / or the confidence score associated with the target data set 126 is associated with the first step, the data range assessment process, and can be used to prevent the target neural network model 128 from being applied to the target data set, or otherwise facilitate adjusting downstream weighting in results generated based on applying the target neural network model 128 to the target data set 126. For example, in some implementations, if the target data set 126 is determined to be outside the range of the training data set 124, the target data set is determined to be associated with a low degree of correspondence with the training data set 124, and / or the target data set 126 is otherwise associated with a low confidence score (e.g., relative to a threshold confidence score), the target data acceptability component 108 may classify the target data set as unsuitable for the target neural network model 128. Based on determining that the target data set is unsuitable for the target neural network model 128, in some embodiments, the target data acceptability component 108 may prevent the target neural network model 128 from being applied to the target data set 126 in the future. In other implementations, based on determining that the target dataset is not suitable for the target neural network model 128, the target data acceptability component may prevent further analysis of the target dataset 126 according to the second step of the two-part process; the model range evaluation process. In another exemplary implementation, for example, if the target dataset 126 is determined to be outside the range of the training dataset 124, the target dataset is determined to be associated with a low degree of correspondence with the training dataset 124, and / or the target dataset 126 is otherwise associated with a low confidence score (e.g., relative to a threshold confidence score), the results generated based on applying the target neural network model 128 to the target dataset 126 may be associated with a warning or weighting indicating a low confidence level of its accuracy level.

[0049] Figure 3 A flow chart of an exemplary, non-limiting process 300 for evaluating and limiting the input data range of a data-driven deep learning model according to one or more embodiments of the subject matter disclosed herein is shown. The method 300 provides a high-level flow chart of the first step (data range evaluation method) of the two-step analysis method described herein. In one or more embodiments, the process 300 can be performed by the data feature extraction component 104, the data analysis component 106, and the target data acceptability component 108. For the sake of brevity, repeated descriptions of similar elements employed in the corresponding embodiments are omitted.

[0050] exist Figure 1 Reference from the perspective of Figure 3According to the process 300, at 302, first training data features 304 (e.g., feature vectors) can be extracted from the training data set 124 (e.g., using the data feature extraction component 104). For example, in one or more embodiments, the data feature extraction component 104 can extract the first training data features 304 using the method 200. At 306, first target data features 308 (e.g., feature vectors) can also be extracted from the target data set 126 (e.g., using the data feature extraction component 104). For example, the data feature extraction component 104 can also extract the first training data features 304 using the method 200. Although the specific methods used to extract the first feature vectors from the training data set and the target data set can vary (e.g., can include other methods instead of AEE), it should be understood that the data feature extraction component 104 uses the same method to extract the first training data features 304 and the first target data features 308.

[0051] At 310, a first degree of correspondence between the first training data feature 304 and the first target data feature 308 may then be determined (e.g., using the data analysis component 106). For example, the data analysis component 106 may employ one or more statistical and / or machine learning based methods (e.g., Z-score analysis for each dimension, Mahalanobis distance analysis for multiple dimensions, t-SNE analysis for lower dimensions to obtain similarity distances, one-class SVM classification for anomaly detection, isolation forests for anomaly detection, etc.) to determine the scope of the training data set 124 based on the first training data feature 304, and further determine the degree of correspondence between the first training data feature 304 and the first target data feature 308. In this regard, depending on the machine learning / analysis method employed, the data analysis component 106 may determine a z-score representing the degree of correspondence between the first training data feature 304 and the first target data feature 308, a distance measure representing the degree of correspondence between the first training data feature 304 and the first target data feature 308, a percentage value representing the degree of correspondence between the first training data feature 304 and the first target data feature 308, etc.

[0052] At 312, the target data acceptability component 108 can then determine whether the first degree of correspondence is acceptable. For example, in some embodiments, the target data acceptability component 108 can determine whether the measurement representing the degree of correspondence meets the acceptability criteria (e.g., minimum threshold, maximum distance, minimum z-score, maximum degree of deviation, etc.). If at 312, the first degree of correspondence is determined to be acceptable, the process 300 can continue to step 2 of the two-part analysis described herein at the model range evaluation at 316. (See below for details.) Figure 4(Discuss model range analysis). However, if it is determined at 312 that the first degree of correspondence is unacceptable, process 300 may proceed to 314, where target data acceptability component 108 determines that applying the target neural network model to the target data set will not generate results with an acceptable level of accuracy and classifies the target data set as unsuitable for use with the target neural network model. In other words, based on the first degree of correspondence being less than an acceptable degree of correspondence, the confidence level in the accuracy of the potential model results generated based on applying the model to the target data set is also deemed unacceptable (e.g., too low).

[0053] For example, in some embodiments, in association with determining whether a first degree of correspondence between a first training data feature 304 and a first target data feature 308 is acceptable, the target data acceptability component 108 may also associate a confidence score representing the degree of correspondence between the first training data feature 304 and the first target data feature 308 with the target data set 126. For example, in some implementations, the confidence score may be a binary value representing acceptable (e.g., within the range of the training data set 124) or unacceptable (e.g., outside the range of the training data set 124). In another embodiment, the confidence score may correspond to the degree of correspondence, such that the higher the degree of correspondence, the higher the confidence score. Using these embodiments, the confidence score associated with the target data set 126 may be used to weight or otherwise adjust downstream results generated based on applying the target neural network model 128 to the target data set.

[0054] See again Figure 1, the second step of the two-step process (the model range assessment step) involves assessing the range of the target neural network model 128 based on applying the target neural network model 128 to the training data set 124. In this regard, the structure of the target neural network model 128, the number of hidden layers, the output parameters generated at each hidden layer, the number of parameters generated, etc. are controlled in part based on the training data used to develop and train the model. The second step of the two-step process can be used to determine whether and with what confidence the DNN model can process the target data set 126 in a manner consistent with the training data set 124 to produce accurate results based on the results generated based on the training data set 124. In various embodiments, the second step in the two-step process is performed only if it is determined that the target data set 126 is within the range of the training data set (if the first target data feature vector exhibits an acceptable degree of correspondence with the first training data feature vector) according to the first step analysis in the two-step process, i.e., the data range assessment step. Utilizing these embodiments, if it is determined that the target data set 126 is associated with an unacceptable degree of correspondence (e.g., relative to a predefined acceptance criterion) with the training data set 124, the target data set 126 can be deemed unsuitable for use with the target neural network model 128, and further analysis of the target data set according to the model evaluation step can be omitted. In one or more embodiments, the model range evaluation step involves the model application component 112, the model feature extraction component 114, the model analysis component 116, and the target data acceptability component 108.

[0055] Based on the model scope assessment, the model application component 112 may apply a subset of layers of the target neural network model 128 to both the training data set 124 and the target data set 126 to generate two sets of model-based features, one set for the training data set and one set for the target data set. The layer subset may include one or more layers of the target neural network model 128 other than the final output layer. For example, the DNN model includes a plurality of intermediate layers between the input layer and the output layer. These intermediate layers may include, for example, hidden layers, convolutional layers, fully connected layers, partially connected layers, etc. These intermediate layers generate output parameters / features that are fed as inputs to subsequent downstream layers. Therefore, the model-based features generated based on applying a subset of DNN model layers (e.g., one or more intermediate layers) to the training data set 124 and the target data set 126 are different from the data-based extracted features / feature vectors extracted and evaluated according to step 1 of the two-step analysis, because the model-based features correspond to the outputs of the various nodes and layers of the target neural network model 128. In this regard, the model-based features take into account the specific scope and structure of the trained DNN model to evaluate and process the training data set 124.

[0056] Based on the model-wide assessment, the same subset of layers of the target neural network model 128 may be applied to both the training data set 124 and the target data set 126 to generate these types of model-based features (e.g., the outputs of the corresponding intermediate layers). The set of model-based features generated based on applying the subset of layers of the target neural network model 128 to the training data set 124 is referred to herein as second training data features or second training data feature vectors. The set of model-based features generated based on applying the subset of layers of the target neural network model 128 to the target data set 126 is referred to herein as second target data features or second target data feature vectors. In one or more embodiments, the model application component 112 may apply a subset of layers (e.g., one or more intermediate layers) of the target neural network model 128 to the training data set 124 to generate a first set of model-based features. The model feature extraction component 114 may also extract second training data features / feature vectors from the first set of model-based features. The model application component 112 may also apply the same subset of layers (e.g., one or more intermediate layers) of the target neural network model 128 to the target data set 126 to generate a second set of model-based features. The model feature extraction component 114 may also extract second target data features / feature vectors from the second set of model-based features.

[0057] Using the same statistical and / or machine learning analysis methods used by the data analysis component 106 in the first step, the model analysis component 116 can further determine or limit the scope of the target neural network model 128 based on the second training data features / feature vectors, and compare the second target data features / feature vectors with the second training data features / feature vectors to determine whether the target data set is within the scope of the target neural network model 128. For example, the analysis methods used by the model analysis component 116 may include, but are not limited to: Z-score analysis for each dimension, Mahalanobis distance analysis on multiple dimensions, t-SNE analysis on lower dimensions to obtain similar distances, one-class SVM classification for anomaly detection, isolation forests for anomaly detection, etc. (regardless of which one the data analysis component 106 uses to determine and evaluate the scope of the training data set 124 relative to the target data set 126).

[0058] In this regard, the model analysis component 116 can employ the same statistical and / or machine learning analysis methods (e.g., one or more of those described above) employed by the data analysis component 106 to compare the second training data features / feature vectors with the second target data features / feature vectors to determine the degree of correspondence between the second training data features / feature vectors and the second target data features / feature vectors. The measured data of the degree of correspondence is in addition to the measured data of the first degree of correspondence determined by the data analysis component 106 according to the first step of the two-step analysis method disclosed herein. For ease of explanation, the degree of correspondence determined by the data analysis component 106 representing the degree of correspondence between the first training data features / feature vectors and the first target features / feature vectors is referred to herein as the "first degree of correspondence", and the degree of correspondence determined by the model analysis component 116 representing the degree of correspondence between the second training data features / feature vectors and the second target features / feature vectors is referred to herein as the "second degree of correspondence".

[0059] As with the data range analysis, depending on the statistical / machine learning analysis method used, the second degree of correspondence may be based on a Z-score, a distance measure, a distribution measure, a percentage value, a combination of two or more similarity measures, etc. In this regard, the specific type of measurement data used to represent the second degree of correspondence (e.g., the degree of correspondence between the second training data feature / feature vector and the second target feature / feature vector) may vary based on the type of statistical and / or machine learning comparative analysis method used. However, in one or more embodiments, because the data analysis component 106 and the model analysis component 116 apply the same statistical / machine learning comparative analysis method, the first degree of correspondence measurement data and the second degree of correspondence measurement data will be the same type of measurement (e.g., the two degrees of correspondence will include a Z-score, a distance measure, a distribution measure, a percentage value, etc.).

[0060] The model acceptability component 118 may further evaluate the second correspondence degree measurement data determined by the model analysis component 116 to determine whether the target data set 126 can be considered within the scope of the target neural network model 128. For example, in some embodiments, the model acceptability component 118 may also adopt a predefined acceptability criterion for the second correspondence degree measurement data. The second predefined acceptability criterion may include, for example, a predefined benchmark or threshold correspondence value (e.g., a minimum Z-score, a maximum degree of deviation, a minimum corresponding percentage value, a maximum degree of separation, etc.). According to these embodiments, the model acceptability component 118 may determine whether the second target data feature / feature vector exhibits an acceptable degree of correspondence with the second training data feature / feature vector, so that the target data set 126 can be considered within the scope of the target neural network model 128 based on whether the second correspondence degree measurement data satisfies the second acceptability criterion. In some embodiments, the second predefined acceptability criterion may be the same as the first predefined acceptability criterion used to evaluate the first correspondence degree (e.g., the same threshold or benchmark value). In other embodiments, the second predefined acceptability criterion and the first predefined acceptability criterion may be different.

[0061] In various embodiments, if the model acceptability component 118 determines that the target data set 126 is outside the scope of the target neural network model 128 (e.g., meaning that a second degree of correspondence is observed between the second training data feature / feature vector and the second target data feature / feature vector that does not satisfy the second acceptability criterion), it can be assumed that the inference results generated based on applying the target neural network model 128 to the target data set 126 may be inaccurate. According to these embodiments, even if the target data set 126 is determined to be within the scope of the training data set 124 (e.g., according to the data range assessment, step-1 of the two-step analysis), if the second target data feature / feature vector is outside the scope of the second training data feature / feature vector, the model acceptability component 118 may associate the target data set 126 with a low confidence score, which represents a low confidence in the accuracy of the results generated based on applying the target neural network model 128 to the target data set. In some embodiments, the confidence score can be a binary score reflecting the final determination that the target data set 126 is suitable or not suitable for the target neural network model 128. In other embodiments, the second confidence score can be non-binary and reflect the degree of correspondence between the second training data features / feature vectors and the second target data features / feature vectors. For example, the second confidence score can increase as the second degree of correspondence increases. In one or more embodiments, based on determining that the target data set 126 is outside the scope of the target neural network model 128 and / or the target data set 126 is associated with an unacceptable confidence score (e.g., relative to a minimum confidence score), the model acceptability component 118 can prevent the target neural network model 128 from being applied to the target data set 126, or otherwise help weight the results generated based on applying the target neural network model 128 to the target data set 126.

[0062] However, if the second target feature is within the scope of the target neural network model 128 (e.g., meaning that the second target data feature / feature vector exhibits an acceptable degree of correspondence with the second training data feature / feature vector), the target data acceptability component 108 may associate a high confidence score with the target neural network model 128, wherein the high confidence score reflects a high confidence that the target neural network model 128 will generate accurate results when applied to the target data set 126. The (second) confidence score may be binary and / or increase as the second degree of correspondence increases. In this regard, based on a first determination at the first step (data evaluation step) that the target data set 126 is within the scope of the training data set 124 and a second determination at the second step (model evaluation step) that the target data set 126 is within the scope of the target neural network model 128, the target data acceptability component 108 may authorize the target data set 126 to be applied to the DNN model. Furthermore, results generated based on applying the DNN model to the target data set may be associated with a high degree of accuracy (e.g., according to a predefined accuracy scale) reflecting a first degree of correspondence and / or a second degree of correspondence (e.g., determined at a data range evaluation step and / or a model range evaluation step).

[0063] In some embodiments, the model acceptability component 118 can determine a cumulative correspondence score and / or confidence score for the target data set 126 that is a combination of a first correspondence degree and a second correspondence degree. For example, if the model acceptability component 118 determines that the first training data feature / vector and the first target data feature / vector exhibit a first correspondence degree of 85%, and the target data acceptability component 108 also determines that the second training data feature / vector and the second target data feature / vector exhibit a second correspondence degree of 95%, the model acceptability component 118 can generate a final "correspondence" or "confidence" score for the target data set 126 that is a function of the combination of the first correspondence value and the second correspondence value (e.g., the average of these two values, which in this example would be 90%). In some specific implementations, the model acceptability component 118 can also determine whether to classify the target data as suitable or unsuitable for the target neural network model 128 based on this final correspondence or confidence score relative to the final acceptability criteria of the cumulative score. With these specific implementations, even if one of the first correspondence measure or the second correspondence measure is low (e.g., relative to a predefined acceptability criterion baseline value), if the other correspondence measure is high enough so that the final cumulative correspondence measure satisfies the final acceptability criterion, the final correspondence or confidence score may still be high enough to classify the target data set as suitable for use in the target neural network model 128. Thus, with these embodiments, even if the first correspondence measure determined in association with the data range assessment is low, such that the target data set 126 is classified as being outside the range of the training data set 124, the system 100 may be configured to proceed with the model range assessment, i.e., step-2 in the two-part analysis.

[0064] Figure 4 A flow chart of an exemplary non-limiting process for evaluating and limiting the scope of model-dependent features of a data-driven deep learning model according to one or more embodiments of the subject matter disclosed herein is shown. Process 400 provides a high-level flow chart of the first step (data range evaluation process) of the two-step analysis process described herein. In one or more embodiments, process 400 can be performed by model application component 112, model feature extraction component 114, model analysis component 116, and model acceptability component 118. For the sake of brevity, repeated descriptions of similar elements employed in the corresponding embodiments are omitted.

[0065] exist Figure 1 Reference from the perspective of Figure 4According to process 400, at 402, a subset of layers of a model (e.g., a target neural network model 128) may be applied to a training data set 124 (e.g., by a model application component 112). At 404, second training data features 406 (e.g., feature vectors) of the training data set 124 may be extracted based on applying the subset of the model layers to the training data set (e.g., using a model feature extraction component 114). For example, in one or more embodiments, the model feature extraction component 114 may extract model parameters generated based on applying the subset of the model layers to the set of training data 122, and generate the second training data features 406 based on the model parameters. At 408, a (same) subset of the layers of the model (e.g., a target neural network model 128) may be applied to a target data set 126 (e.g., by a model application component 112). At 410, second target data features 412 (e.g., feature vectors) generated based on applying the same subset of the model layers to the target data set 126 may be extracted (e.g., using a model feature extraction component 114). For example, in one or more embodiments, the model feature extraction component 114 can extract model parameters generated based on applying a subset of the model layers to the target data set 126, and generate a second training target feature vector 406 based on these model parameters.

[0066] At 414, a second degree of correspondence may then be determined (e.g., by model analysis component 116) between second training data features 406 and second target data features 412 (e.g., using Z-score analysis for each dimension, Mahalanobis distance analysis for multiple dimensions, t-SNE analysis for lower dimensions to obtain similarity distances, one-class SVM classification for anomaly detection, isolation forests for anomaly detection, etc.). For example, model analysis component 116 may employ the same one or more statistical and / or machine learning-based methods used by data analysis component 106 to determine a first degree of correspondence between a first training feature vector and a first target feature vector according to a first step in a two-step analysis; data range assessment (e.g., process 300). At 416, model acceptability component 118 may then determine whether the second degree of correspondence is acceptable. For example, in some embodiments, model acceptability component 118 may determine whether a measure representing the degree of correspondence meets an acceptability criterion (e.g., a minimum threshold, a maximum distance, a minimum z-score, a maximum degree of deviation, etc.).

[0067] If at 416, it is determined that the second degree of correspondence is unacceptable, process 400 may proceed to 418, where model acceptability component 118 determines that applying the target neural network model to the target data set will not generate results with an acceptable level of accuracy and classifies the target data set as unsuitable for the target neural network model. In this regard, based on the second level of accuracy being unacceptable (e.g., failing to meet defined acceptability criteria), the confidence level in the accuracy of the potential model results generated based on applying the target neural network model to the target data set 126 will also be unacceptable (e.g., too low). However, if at 416, the second degree of correspondence is determined to be acceptable, process 400 may continue to 420, where model acceptability component 118 determines that applying the target neural network model to the target data set will generate results with an acceptable level of accuracy and classifies the target data set as suitable for the target neural network model.

[0068] In some embodiments, in association with determining whether the second degree of correspondence between the second training data feature 406 and the second target data feature 412 is acceptable, the model acceptability component 118 may also associate a (second) confidence score representing the (second) degree of correspondence between the second training data feature 406 and the second target data feature 412 with the target data set 126. For example, in some implementations, the (second) confidence score may be a binary value representing acceptable (e.g., within the range of the target neural network model 128) or unacceptable (e.g., outside the range of the target neural network model 128). In another embodiment, the (second) confidence score may correspond to the (second) degree of correspondence, such that the higher the degree of correspondence, the higher the confidence score. Using these embodiments, the (second) confidence score associated with the target data set 126 may be used to weight or otherwise adjust downstream results generated based on applying the target neural network model 128 to the target data set.

[0069] For example, in some implementations, the target dataset 126 may be associated with a first confidence score that indicates a first degree of correspondence between the first training feature vector and the first target feature vector (e.g., according to process 300). The target dataset 126 may also be associated with a second confidence score that indicates a second degree of correspondence between the second training data feature 406 and the second target data feature (e.g., according to process 400). The first confidence score and / or the second confidence score may be used to control the application of the target neural network model 128 to the target dataset 126 of the domain and / or to weight the value of the result generated based on the application of the target neural network model 128 to the target dataset 126. For example, in some implementations, if the first confidence score and / or the second confidence score, or a cumulative / final confidence score based on a combination of the first confidence score and the second confidence score, is less than a predefined confidence score benchmark or threshold, the model application component 112 may be configured to prevent the target neural network model 128 from being applied to the target dataset 126 of the domain. In another embodiment, system 100 (or another system) may choose to apply target neural network model 128 to target data set 126 regardless of the final / cumulative confidence score associated with target neural network model 128, but weight the accuracy of the results according to the cumulative / final confidence score.

[0070] Figure 5 A block diagram of another exemplary non-limiting system 500 that facilitates evaluation and scope of a data-driven deep learning model according to one or more embodiments of the disclosed subject matter is shown. System 500 includes features that are the same or similar to system 100, with the addition of a notification component 502, an evaluation component 506, a target dataset evaluation notification 504, an evaluation report 508, and training data scope and model scope information 510. For the sake of brevity, repeated descriptions of similar elements employed in various embodiments are omitted.

[0071] In one or more embodiments, notification component 502 can generate a notification or warning indicating that target data set 126 is determined to be associated with a correspondence program that is less than an acceptable degree of correspondence, is associated with a low confidence score, and / or is otherwise outside the scope of training data set 124 (e.g., according to the first step in a two-step analysis; data range assessment, process 300); and / or target neural network model 128 (e.g., according to the second step in a two-step analysis; model range assessment, process 400). In the illustrated embodiment, the notification is identified as target data set assessment notification 504. The notification or warning can be used to prevent the target neural network model 128 from being applied to the target data set (e.g., by model application component 112), or to otherwise adjust the downstream weighting of inferences derived from applying the target neural network model 128 to the target data.

[0072] For example, in some implementations, if the target data acceptability component 108 determines that the target data set 126 is outside the range of the training data set 124 associated with the data range assessment (e.g., based on a first degree of correspondence between the first target data feature / feature vector and the first training data feature / feature vector being less than an acceptable degree of correspondence), the notification component 502 can generate a target data set assessment notification 504 for presentation or otherwise to an entity responsible for applying the target neural network model 128 (e.g., the model application component 112, a user interacting with software that invokes the target neural network model 128 to generate inferences based on the target data set 126, or other suitable entity). The notification can include information indicating that the target data set 126 is outside the range of the training data set 124. In some implementations, in addition to indicating that the target data set 126 is within or outside the range of the training data set 124, the notification can also include information such as the first degree of correspondence, the acceptability criteria, a determined first confidence level in the target data set 126 reflecting the first degree of correspondence, a level of predicted accuracy of potential results generated based on the target data set 126, and the like. Based on receiving the notification, the entity may decide at its discretion whether to continue applying the target neural network model 128 to the target data set 126 or to initiate an alternative (e.g., a manual, user-assisted process, or other review process) to evaluate the target data set 126. For example, if the target data set includes medical images, and the target neural network model 128 is configured to generate an automatic diagnosis based on the input medical images, then based on receiving the notification that the target data set 126 is outside the range of the training data set 124, the entity may assume that the diagnosis result based on applying the target neural network model 128 to the target data set 126 has a relatively high probability of inaccuracy (e.g., where the probability is based on a first confidence associated with the target data set 126). Therefore, in addition to and / or alternatively to having the target neural network model 128 generate a diagnosis based on the image, the entity may direct the attending physician or a group of expert physicians to manually review the input medical images. In some embodiments, even if the target data set 126 is determined to be within the range of the training data set 124, the notification component 502 may generate a notification including such information about the results of the data range analysis step after the data range analysis step.

[0073] Notification component 502 may similarly generate target dataset evaluation notification 504 following model range analysis of the results of the model range analysis step. For example, if target data acceptability component 108 determines that target dataset 126 is outside the scope of target neural network model 128 associated with the model range evaluation (e.g., based on second target data features / feature vectors exhibiting a second degree of correspondence with second training data features / feature vectors that is less than an acceptable degree of correspondence), notification component 502 may generate target dataset evaluation notification 504 for presentation or otherwise to an entity responsible for applying target neural network model 128 (e.g., model application component 112, a user interacting with software that invokes target neural network model 128 to generate inferences based on target dataset 126, or other suitable entity). The notification may include information indicating that target dataset 126 is outside the scope of target neural network model 128, identifying the determined second degree of correspondence, acceptability criteria for evaluating the second degree of correspondence, a determined second level of confidence in target dataset 126 reflecting the second degree of correspondence, information about a level of predictive accuracy of potential results generated based on target dataset 126, and the like. Based on receipt of the notification, the entity can decide at its discretion whether to proceed with applying the target neural network model 128 to the target data set 126 or to initiate an alternative (e.g., a manual, user-assisted process, or other review process) to evaluate the target data set 126.

[0074] Evaluation component 506 may be configured to generate and provide an evaluation report 508 regarding the results of a data range evaluation and / or a model range evaluation applied to a particular target data set 126. For example, the evaluation report may include the same type of information regarding target data set 126 that may be included in target data set evaluation notification 504. However, the evaluation report may be generated even if target data set 126 is determined to be within the scope of training data set 124 and / or within the scope of target neural network model 128. In some embodiments, evaluation report 508 may be used by an appropriate entity to determine whether to apply target neural network model 128 to target data set 126 and / or to determine how to consume and / or weight the results generated based on applying target neural network model 128 to target data set 126 (e.g., model application component 112, a user, another system that applies target neural network model 128, another component that provides a recommendation regarding whether to apply target neural network model 128, another component that evaluates and / or applies the results of target neural network model 128, etc.).

[0075] Evaluation component 506 may also be configured to generate and collate range information generated by data analysis component 106 regarding the determined range of training data set 124 and the determined range of target neural network model 128 determined by model analysis component 116. This "range information" may be collected in a suitable network-accessible database and identified in system 500 as training data range and model range information 510. For example, range information may include information about a particular training data set 124 used to develop and train the target neural network model 128, identifying extracted first training data features / feature vectors of the training data set 124. Range information may also include information defining the range of the training data set 124 based on the first training data features / feature vectors. Range information may similarly include information identifying second training data features / feature vectors of the target neural network model 128 and / or the range of the target neural network model 128 determined based on the second training data features / feature vectors. Computing device 102 may use this range information to evaluate and process future additional target data sets with increased computation time / processing speed. For example, the data feature extraction component 104 may only extract new first target data feature vectors for the new target data set because the data analysis component 106 may perform its analysis by reusing the previously extracted first training data features included in the training data range and model range information 510. Similarly, the model application component 112 need only apply a subset of the model layers to the new target data set, and the model feature extraction component 114 need only extract new second target data features therefrom because the model analysis component 116 may reuse the initial second training data features included in the training data range and model range information 510. In this regard, each time the computing device receives 102 new data for potential application of the target neural network model 128, the computing device 102 may reuse the previously determined training data range information and target neural network model 128 range information, thereby enhancing the processing time required to evaluate the new target data set.

[0076] Figure 6 A flowchart of an exemplary non-limiting computer-implemented method 600 that facilitates evaluating and limiting the scope of a data-driven deep learning model according to one or more embodiments of the disclosed subject matter is provided. For the sake of brevity, repeated descriptions of similar elements employed in various embodiments are omitted.

[0077] At 602, a system (e.g., system 100, system 500, etc.) operably coupled to a processor may be an AAE to identify and extract a first training data feature included in a training data set and a first target data feature included in a target data set (e.g., using data feature extraction component 104). At 604, the system may determine a first degree of correspondence between the training data set and the target data set based on an analysis of the correspondence between the first training data feature and the first target data feature using a statistical analysis method or a machine learning analysis method (e.g., using data analysis component 106). At 606, the system may determine whether to continue applying a target neural network model to the target data based on the first degree of correspondence (e.g., using target data acceptability component 108), wherein the target neural network model is trained on the training data set.

[0078] Figure 7 Another exemplary non-limiting computer-implemented method 700 flow chart is provided that facilitates evaluating and limiting the scope of a data-driven deep learning model according to one or more embodiments of the disclosed subject matter. For the sake of brevity, repeated descriptions of similar elements employed in various embodiments are omitted.

[0079] At 702, a system (e.g., system 100, system 500, etc.) operably coupled to a processor may employ an adversarial autoencoder to identify and extract training data features included in a training data set and target data features included in a target data set, wherein a target neural network model is trained on the training data set (e.g., using data feature extraction component 104). At 704, the system may determine a degree of correspondence between the training data set and the target data set based on an analysis of the correspondence between the training data features and the target data features using a statistical analysis method or a machine learning analysis method (e.g., using data analysis component 106). At 706, the system may determine that applying the target neural network model to the target data set will generate a result with an unacceptable level of accuracy based on the degree of correspondence failing to meet a threshold degree of correspondence (e.g., using target data acceptability component 108). At 708, the system may classify the target data set as unsuitable for the target neural network model based on determining that applying the target neural network model to the target data set will generate a result with an unacceptable level of accuracy.

[0080] Figure 8 A flowchart of another exemplary non-limiting computer-implemented method 800 that facilitates evaluating and limiting the scope of a data-driven deep learning model according to one or more embodiments of the disclosed subject matter is provided. For the sake of brevity, repeated descriptions of similar elements employed in various embodiments are omitted.

[0081] At 802, a system (e.g., system 100, system 500, etc.) operably coupled to a processor may employ an adversarial autoencoder to identify and extract training data features included in a training data set and target data features included in a target data set, wherein a target neural network model is trained on the training data set (e.g., using data feature extraction component 104). At 804, the system may determine a degree of correspondence between the training data set and the target data set based on an analysis of the correspondence between the training data features and the target data features using a statistical analysis method or a machine learning analysis method (e.g., using data analysis component 106). At 806, the system may determine that the target data set is outside the range of the training data set based on the degree of correspondence failing to meet a threshold degree of correspondence (e.g., using target data acceptability component 108), wherein based on the target data set being determined to be outside the range of the training data set, the system determines that applying the target neural network model to the target data set will generate a result with an unacceptable level of accuracy. At 808, the system may generate a notification regarding the unacceptable level of accuracy before applying the target data set to the target neural network model.

[0082] Fig. 9 A flowchart of another exemplary non-limiting computer-implemented method 900 that facilitates evaluating and limiting the scope of a data-driven deep learning model according to one or more embodiments of the disclosed subject matter is provided. For the sake of brevity, repeated descriptions of similar elements employed in various embodiments are omitted.

[0083] At 902, a system operably coupled to a processor (e.g., system 100, system 500, etc.) may extract a first training data feature included in a training data set and a first target data feature included in a target data set (e.g., using data feature extraction component 104). At 904, the system may determine a first degree of correspondence between the first training data feature and the first target data feature (e.g., using data analysis component 106). At 906, based on determining that the first degree of correspondence is acceptable (e.g., based on defining acceptability criteria), the system may further extract a second training data feature generated based on applying a subset of layers of a target neural network model to the training data set, and extract a second target data feature generated based on applying the subset of layers to the target data set (e.g., using target data acceptability component 108, model application component 112, and model feature extraction component 114). At 908, the system may determine a second degree of correspondence between the second training data feature and the second target data feature (e.g., using model analysis component 116). At 910, the system can further determine whether applying the target data set to the target neural network model will generate results with an acceptable level of accuracy based on the second degree of correspondence (e.g., using target data acceptability component 108).

[0084] Fig.10 A flowchart of another exemplary non-limiting computer-implemented method 1000 that facilitates evaluating and limiting the scope of a data-driven deep learning model according to one or more embodiments of the presently disclosed subject matter is provided. For the sake of brevity, repeated descriptions of similar elements employed in various embodiments are omitted.

[0085] At 1002, a system operably coupled to a processor (e.g., system 100, system 500, etc.) may employ a machine learning model (e.g., AAE) to extract a first training data feature included in a training data set and a first target data feature included in a target data set (e.g., using data feature extraction component 104). At 1004, the system may determine whether the target data set is within a defined data range of the training data set based on an analysis of the correspondence between the first training data feature and the first target data feature (e.g., using data analysis component 106 and target data acceptability component 108). At 1006, based on determining that the target data set is within the defined data range of the training data set, the system may also extract a second training data feature generated based on applying a subset of layers of a target neural network model to the training data set (e.g., via model application component 112 and model feature extraction component 114). At 1008, the system may extract a second target data feature generated based on applying a subset of layers of a target neural network model to the target data set (e.g., via model application component 112 and model feature extraction component 114). At 1010, the system can determine whether the target data set is within the defined model range of the target neural network model based on the analysis of the correspondence between the second training data features and the second target data features. At 1012, the system can also determine whether applying the target data set to the target neural network model will generate results with an acceptable level of accuracy based on whether the target data set is within the defined range of the target neural network model (e.g., using target data acceptability component 108).

[0086] One or more embodiments may be systems, methods and / or computer program products at any possible level of technical detail of integration. The computer program product may include a computer-readable storage medium (or multiple media) having computer-readable program instructions thereon for causing a processor to perform aspects of the present invention.

[0087] A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following items: a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device (such as a punch card or a raised structure in a groove on which instructions are recorded), and any suitable combination of the foregoing items. As used herein, a computer-readable storage medium should not be understood to be a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted by a wire.

[0088] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The network can include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network, and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the corresponding computing / processing device.

[0089] The computer-readable program instructions for performing the operation of the present invention can be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, configuration data of integrated circuits, or source code or object code written in any combination of one or more programming languages ​​(including object-oriented programming languages, such as Smalltalk, C++, etc.) and process programming languages ​​(such as "C" programming language or similar programming languages). Computer-readable program instructions can be executed completely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or completely on a remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network (including local area network (LAN) or wide area network (WAN)), or can be connected to an external computer (for example, by using the Internet of an Internet service provider). In some embodiments, the electronic circuit including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer-readable program instructions to personalize the electronic circuit by utilizing the state information of the computer-readable program instructions, so as to perform aspects of the present invention.

[0090] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present invention. It is understood that each frame of the flowchart illustration and / or block diagram, and the combination of frames in the flowchart illustration and / or block diagram can be implemented by computer-readable program instructions.

[0091] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create a device for implementing the function / action specified in one or more boxes of the flow chart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which can guide the computer, programmable data processing device, and / or other equipment to function in a specific manner, so that the computer-readable storage medium with instructions stored therein includes an article of manufacture, which includes instructions for implementing various aspects of the function / action specified in one or more boxes of the flow chart and / or block diagram.

[0092] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be executed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, such that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0093] The flow charts and block diagrams in the accompanying drawings illustrate the possible specific implementation architecture, functions and operations of the system, method and computer program product according to various embodiments of the present invention. In this regard, each frame in the flow chart or block diagram can represent a module, a fragment or a part of an instruction, which includes one or more executable instructions for implementing a specified logical function. In some alternative specific implementations, the functions indicated in the frame may not occur in the order indicated in the figure. For example, depending on the functions involved, two frames shown in succession can actually be executed substantially simultaneously, or sometimes these frames may be executed in reverse order. It should also be noted that each frame of the block diagram and / or flow chart diagram and the combination of frames in the block diagram and / or flow chart diagram can be implemented by a system based on dedicated hardware that performs a specific function or action or performs a combination of dedicated hardware and computer instructions.

[0094] Combination Fig.10 , the systems and processes described below may be embodied in hardware, such as a single integrated circuit (IC) chip, multiple ICs, application specific integrated circuits (ASICs), etc. In addition, the order in which some or all of the process blocks appear in each process should not be considered limiting. Instead, it should be understood that some process blocks can be performed in a variety of orders, not all of which can be explicitly shown herein.

[0095] refer to Fig.11 , an exemplary environment 1100 for implementing various aspects of the claimed subject matter includes a computer 1102. The computer 1102 includes a processing unit 1104, a system memory 1106, a codec 1135, and a system bus 1108. The system bus 1108 couples system components including, but not limited to, the system memory 1106 to the processing unit 1104. The processing unit 1104 can be any of a variety of available processors. Dual microprocessors and other multi-processor architectures can also be used as the processing unit 1104.

[0096] The system bus 1108 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus or external bus, or a local bus using a variety of available bus architectures including, but not limited to, Industry Standard Architecture (ISA), Micro Channel Architecture (MSA), Extended ISA (EISA), Intelligent Drive Electronics (IDE), VESA Local Bus (VLB), Peripheral Component Interconnect (PCI), card bus, Universal Serial Bus (USB), Advanced Graphics Port (AGP), Personal Computer Memory Card International Association bus (PCMCIA), FireWire (IEEE 1394), and Small Computer System Interface (SCSI).

[0097] In various embodiments, the system memory 1106 includes a volatile memory 1110 and a non-volatile memory 1112, which may employ one or more of the disclosed memory architectures. A basic input / output system (BIOS), which contains the basic routines for transferring information between elements within the computer 1102, such as during startup, is stored in the non-volatile memory 1112. In addition, according to the innovation of the present invention, the codec 1135 may include at least one of an encoder or a decoder, wherein at least one of the encoder or the decoder may be composed of hardware, software, or a combination of hardware and software.

[0098] Although the codec 1135 is shown as a separate component, the codec 1135 can be included in the non-volatile memory 1112. By way of example and not limitation, the non-volatile memory 1112 may include a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, a 3D flash memory, or a resistive memory, such as a resistive random access memory (RRAM). In at least some embodiments, the non-volatile memory 1112 may employ one or more of the disclosed memory devices. In addition, the non-volatile memory 1112 may be a computer memory (e.g., physically integrated with the computer 1102 or its motherboard) or a removable memory. Examples of suitable removable memories that can be used to implement the disclosed embodiments may include a secure digital (SD) card, a compact flash (CF) card, a universal serial bus (USB) memory stick, and the like. The volatile memory 1110 includes a random access memory (RAM) that acts as an external cache memory, and one or more disclosed memory devices may also be employed in various embodiments. By way of example and not limitation, RAM can be provided in a variety of forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), and enhanced SDRAM (ESDRAM).

[0099] The computer 1102 may also include removable / non-removable, volatile / nonvolatile computer storage media. Fig.11 For example, a disk storage device 1114 is shown. The disk storage device 1114 includes, but is not limited to, devices such as disk drives, solid-state disks (SSDs), flash memory cards, or memory sticks. In addition, the disk storage device 1114 can include storage media alone or in combination with other storage media, including but not limited to optical disk drives, such as optical disk ROM devices (CD-ROMs), CD recordable drives (CD-R drives), CD rewritable drives (CD-RW drives), or digital versatile disk ROM drives (DVD-ROMs). In order to facilitate the connection of the disk storage device 1114 to the system bus 1108, a removable or non-removable interface, such as an interface 1116, is generally used. It should be understood that the disk storage device 1114 can store information related to the user. Such information can be stored at a server or provided to an application running on a server or user device. In one embodiment, the user can be notified (for example, through one or more output devices 1136) of the type of information stored in the disk storage device 1114 or transmitted to the server or application. The user may be provided an opportunity to opt-in or opt-out of the collection or sharing of such information by the server or application (eg, via input from one or more input devices 1128 ).

[0100] It should be understood that Fig.11 Software is described that acts as an intermediary between a user and the basic computer resources described in a suitable operating environment 1100. Such software includes an operating system 1118. The operating system 1118, which may be stored on disk storage 1114, is used to control and allocate the resources of the computer 1102. Application programs 1120 utilize the operating system 1118 through program modules 1124 to manage resources, as well as program data 1126 stored on system memory 1106 or disk storage 1114, such as boot / shutdown transaction tables, etc. It should be understood that the claimed subject matter can be implemented with various operating systems or combinations of operating systems.

[0101] The user enters commands or information into the computer 1102 through input devices 1128. Input devices 1128 include, but are not limited to, pointing devices such as a mouse, trackball, stylus, touchpad, keyboard, microphone, joystick, gamepad, satellite dish, scanner, TV tuner card, digital camera, digital video camera, webcam, etc. These and other input devices are connected to the processing unit 1104 through the system bus 1108 via interface ports 1130. Interface ports 1130 include, for example, serial ports, parallel ports, game ports, and universal serial bus (USB). Output devices 1136 use some of the same types of ports as input devices 1128. Thus, for example, a USB port can be used to provide input to the computer 1102 and output information from the computer 1102 to output devices 1136. Output adapters 1134 are provided to illustrate that there are some output devices 1136 such as monitors, speakers, and printers, as well as other output devices 1136 that require special adapters. By way of example and not limitation, output adapters 1134 include video and sound cards that provide connections between output devices 1136 and system bus 1108. It should be noted that other devices or systems of devices provide input and output capabilities, such as remote computer 1138.

[0102] The computer 1102 can operate in a networked environment using logical connections to one or more remote computers, such as remote computers 1138. The remote computer 1138 can be a personal computer, a server, a router, a network PC, a workstation, a microprocessor-based device, a peer device, a smart phone, a tablet computer, or other network node, and typically includes many of the elements described relative to the computer 1102. For the purpose of simplicity, only a memory storage device 1140 is shown for the remote computer 1138. The remote computer 1138 is logically connected to the computer 1102 through a network interface 1142 and then connected via a communication connection 1144. The network interface 1142 includes a wired or wireless communication network, such as a local area network (LAN) and a wide area network (WAN) and a cellular network. LAN technologies include fiber distributed data interface (FDDI), copper distributed data interface (CDDI), Ethernet, token ring, etc. WAN technologies include, but are not limited to, point-to-point links, circuit switching networks such as integrated services digital networks (ISDN) and variations thereon, packet switching networks, and digital subscriber lines (DSL).

[0103] The communication connection 1144 refers to the hardware / software used to connect the network interface 1142 to the bus 1108. Although the communication connection 1144 is shown within the computer 1102 for clarity of illustration, the communication connection can also be external to the computer 1102. For example purposes only, the hardware / software required to connect to the network interface 1142 includes internal and external technologies, such as modems, including conventional telephone grade modems, cable modems and DSL modems, ISDN adapters, as well as wired and wireless Ethernet cards, hubs and routers.

[0104] Although the subject matter has been described above in the general context of computer executable instructions of a computer program product running on one and / or multiple computers, it will be appreciated by those skilled in the art that the present disclosure may also or may be implemented in combination with other program modules. Typically, program modules include routines, programs, components, data structures, etc. that perform specific tasks and / or implement specific abstract data types. In addition, it will be appreciated by those skilled in the art that the computer implementation method of the present invention may be practiced with other computer system configurations, including single-processor or multi-processor computer systems, small computing devices, large computers, and computers, handheld computing devices (e.g., PDAs, phones), microprocessor-based or programmable consumer or industrial electronic devices, etc. The illustrated aspects may also be practiced in a distributed computing environment, in which tasks are performed by a remote processing device linked by a communication network. However, some (if not all) aspects of the present disclosure may be practiced on a stand-alone computer. In a distributed computing environment, program modules may be located in local and remote memory storage devices.

[0105] As used in this application, the terms "component", "system", "platform", "interface", etc. may refer to and / or may include computer-related entities, or entities related to an operating machine with one or more specific functions. Entities disclosed herein may be hardware, a combination of hardware and software, software, or software in execution. For example, a component may be, but is not limited to, a process, a processor, an object, an executable file, a thread of execution, a program, and / or a computer running on a processor. By way of example, both the application program and the server running on the server may be components. One or more components may reside in a process and / or a thread of execution, and a component may be located on a computer and / or distributed between two or more computers. In another example, the corresponding component may be executed according to various computer-readable media having various data structures stored thereon. Components may communicate via local and / or remote processes, such as according to a signal with one or more data packets (e.g., data from a component that interacts with another component in a local system, a distributed system, and / or a network (such as, via the Internet with other systems via a signal). As another example, a component may be a device having a specific functionality provided by a mechanical part operated by an electrical or electronic circuit that is operated by a software or firmware application executed by a processor. In this case, the processor may be internal or external to the device and may execute at least a portion of the software or firmware application. As yet another example, a component may be a device that provides a specific functionality through electronic components rather than mechanical parts, where the electronic components may include a processor or other device for executing software or firmware that at least partially imparts functionality to the electronic components. In one aspect, the component may simulate the electronic component, for example, via a virtual machine within a cloud computing system.

[0106] In addition, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless otherwise specified or clear from the context, "X employs A or B" is intended to mean any natural inclusive permutation. That is, if X employs A; X employs B; or X employs both A and B, then "X employs A or B" is satisfied in any of the foregoing cases. In addition, the articles "one" and "an" used in this specification and the drawings should generally be interpreted to mean "one or more" unless otherwise specified or clear from the context to refer to the singular form. As used herein, the terms "example" and / or "exemplary" are used to indicate use as an example, instance, or illustration, and are intended to be non-limiting. For the avoidance of doubt, the subject matter disclosed herein is not limited to such examples. In addition, any aspect or design described herein as "example" and / or "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects or designs, nor is it meant to exclude equivalent exemplary structures and techniques known to those of ordinary skill in the art.

[0107] As used in this subject specification, the term "processor" may refer to substantially any computing processing unit or device, including but not limited to a single-core processor; a single processor with software multithreaded execution capability; a multi-core processor; a multi-core processor with software multithreaded execution capability; a multi-core processor with hardware multithreading technology; a parallel platform; and a parallel platform with distributed shared memory. In addition, a processor may refer to an integrated circuit, an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic controller (PLC), a complex programmable logic device (CPLD), a discrete gate or transistor logic component, a discrete hardware component, or any combination thereof designed to perform the functions described herein. In addition, the processor may utilize a nanoscale architecture (such as, but not limited to, transistors, switches, and gates based on molecules and quantum dots) in order to optimize space usage or enhance the performance of user devices. The processor may also be implemented as a combination of computing processing units. In the present disclosure, terms such as "storage", "storage device", "data storage", "data storage device", "database", and substantially any other information storage component related to the operation and function of the component are used to refer to a "memory component", an entity embodied in the "memory", or a component including a memory. It should be appreciated that the memory and / or memory components described herein may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. By way of example and not limitation, non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), flash memory, or non-volatile random access memory (RAM) (e.g., ferroelectric RAM (FeRAM)). For example, volatile memory may include RAM, which may act as an external cache memory. By way of example and not limitation, RAM may be provided in a variety of forms, such as synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), direct Rambus RAM (DRRAM), direct Rambus dynamic RAM (DRDRAM) and Rambus dynamic RAM (RDRAM). In addition, the memory components of the system or computer-implemented method disclosed herein are intended to include, but are not limited to, including these and any other suitable types of memory.

[0108] What has been described above only includes examples of systems and computer-implemented methods. Of course, it is not possible to describe every conceivable combination of components or computer-implemented methods for the purpose of describing the present disclosure, but those of ordinary skill in the art will recognize that many further combinations and permutations of the present disclosure are possible. In addition, to the extent that the terms "including", "having", "having", etc. are used in the specific embodiments, claims, appendices, and drawings, such terms are intended to be inclusive in a manner similar to the term "including", such as interpreting "including" when used as a transitional word in the claims. Descriptions of various embodiments have been given for illustrative purposes, but these descriptions are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations are obvious to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, the practical application or technical improvement of the technology found on the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A system comprising: a memory storing computer executable components; and a processor that executes the computer executable components stored in the memory, wherein the computer executable components include: A data feature extraction component, wherein the data feature extraction component uses an adversarial autoencoder to identify and extract a first training data feature included in the training data set and a first target data feature included in the target data set; a data analysis component, which determines a first degree of correspondence between the training data set and the target data set based on a comparison between the first training data feature and the first target data feature; and A target data acceptability component is configured to determine whether to continue applying a target neural network model to the target data set based on the first degree of correspondence, wherein the target neural network model is trained on the training data set.

2. The system according to claim 1, wherein the data analysis component limits the scope of the training data set based on the training data feature, and determines the first degree of correspondence based on whether the target data feature is within the scope of the training data set. 3 . The system of claim 1 , wherein the first degree of correspondence indicates a level of predictive accuracy of a result generated based on the application. 4 . The system according to claim 1 , wherein the data analysis component determines the degree of correspondence using a statistical analysis method and a machine learning method.

5. The system of claim 1 , wherein the target data acceptability component determines whether to continue applying the target neural network model to the target data set based on a first defined acceptability criterion for the first degree of correspondence.

6. A system according to claim 5, wherein based on determining that the first degree of correspondence fails to meet the first defined acceptability criteria, the target data acceptability component determines that applying the target neural network model to the target data set will not generate results with an acceptable level of accuracy, and classifies the target data set as unsuitable for use with the target neural network model.

7. The system of claim 5, wherein the computer executable component further comprises: a model application component that applies a subset of layers of the target neural network model to the training data set and the target data set based on a first determination that the first degree of correspondence satisfies the first defined acceptability criterion; and A model feature extraction component extracts second training data features generated based on applying the layer subset to the training data set, and extracts second target data features generated based on applying the layer subset to the target data set.

8. The system of claim 7, wherein the computer executable components further comprise: a model analysis component, the model analysis component determining a second degree of correspondence between the second training data feature and the second target data feature; and A model acceptability component is configured to determine whether applying the target neural network model to the target data set will generate results with an acceptable level of accuracy based on the second degree of correspondence.

9. The system of claim 8, wherein the model analysis component limits the scope of the target neural network model based on the second training data feature, and determines the second degree of correspondence based on whether the second target data feature is within the scope of the target neural network model.

10. The system according to claim 8, wherein the model analysis component determines the second degree of correspondence using a statistical analysis method or a machine learning method.

11. The system according to claim 8, wherein the data analysis component and the model analysis component use the same statistical analysis method or the same machine learning method to respectively determine the first degree of correspondence and the second degree of correspondence.

12. The system of claim 8, wherein the model acceptability component determines that applying the target neural network model to the target data set will not generate the result with the acceptable level of accuracy based on a second determination that the second degree of correspondence fails to satisfy a second defined acceptability criterion.

13. The system of claim 8, wherein the model acceptability component determines whether the second degree of correspondence satisfies the acceptability criterion for the second degree of correspondence based on a second Determining that applying the target neural network model to the target data set will not generate the results with the acceptable level of accuracy.

14. A computer-implemented method comprising: employing, by a system operably coupled to the processor, an adversarial autoencoder to identify and extract first training data features included in the training data set and first target data features included in the target data set; The system determines a first degree of correspondence between the training data set and the target data set based on an analysis of the correspondence between the first training data feature and the first target data feature using a statistical analysis method or a machine learning analysis method; as well as The system determines whether to continue applying a target neural network model to the target data set based on the first degree of correspondence, wherein the target neural network model is trained on the training data set.

15. The computer-implemented method of claim 14, wherein the first degree of correspondence indicates a level of predictive accuracy of a result generated based on the application.

16. The computer-implemented method of claim 14, further comprising: Based on determining that the first degree of correspondence does not satisfy a threshold degree of correspondence, determining, by the system, that applying the target neural network model to the target data set will not produce results with an acceptable level of accuracy; as well as The target data set is classified by the system as being unsuitable for the target neural network model.

17. The computer-implemented method of claim 14, further comprising: Based on a first determination that the first degree of correspondence satisfies a first threshold degree of correspondence, extracting, by the system, second training data features generated based on applying a subset of layers of the target neural network model to the training data set; as well as Second target data features generated based on applying the subset of layers of the target neural network model to the target data set are extracted by the system.

18. The computer-implemented method of claim 17, further comprising: determining, by the system, a second degree of correspondence between the second training data feature and the second target data feature; as well as The system determines whether applying the target neural network model to the target data set will generate results with an acceptable level of accuracy based on whether the second degree of correspondence satisfies a second threshold degree of correspondence.

19. A machine-readable storage medium comprising executable instructions that, when executed by a processor, facilitate performance of operations comprising: Using a machine learning model to extract a first training data feature included in the training data set and a first target data feature included in the target data set; Determining whether the target data set is within a limited data range of the training data set based on an analysis of the correspondence between the first training data feature and the first target data feature; as well as Based in part on whether the target data set is within the defined data range of the training data set, determining whether applying the target data set to a target neural network model developed using the training data set will generate results with an acceptable level of accuracy.

20. The machine-readable storage medium of claim 19, wherein based on determining that the target data set is within the defined data range of the training data set, the operation further comprises: extracting second training data features generated based on applying a subset of layers of the target neural network model to the training data set; extracting a second target data feature generated based on applying the subset of layers of the target neural network model to the target data set; Determining whether the target data set is within a defined model range of the target neural network model based on an analysis of the correspondence between the second training data features and the second target data features; as well as Based on whether the target data set is within the defined range of the target neural network model, it is further determined whether applying the target data set to the target neural network model will generate the result with the acceptable level of accuracy.

Images