Method, device and electronic equipment for automated diagnosis of data

Abstract

The present invention provides a method, apparatus, and electronic device for automated data diagnosis. In this embodiment, driven by a first image set, the model training process is fully automated for data diagnosis, improving model performance. Furthermore, this embodiment automatically adjusts the training plan based on the distribution of characteristic data in the training and test sets, as well as automatically analyzing potential model issues, without the need for manual intervention.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to methods, devices, and electronic equipment for automated diagnosis of data. Background Art

[0002] In current artificial intelligence (AI) applications, data serves as the input for developing deep learning models, and its quality determines the ultimate effectiveness of the deep learning models. However, in the actual model development process, problems such as low data quality, substandard calibration, and inconsistent distribution of training data and test data for model testing often arise, resulting in suboptimal deep learning model results. Summary of the Invention

[0003] The embodiments of the present application provide a method, apparatus, and electronic device for automatically diagnosing data to improve model performance by automatically diagnosing data during model training.

[0004] An embodiment of the present application provides a method for automatically diagnosing data, which is applied to an electronic device and includes:

[0005] Selecting images that meet the model training requirements from the first image set to obtain a second image set; classifying the images in the second image set to obtain a training set and a test set; the training set and the test set have at least one common feature type;

[0006] Determining a target feature type; the feature data distribution of the training set under the target feature type and the feature data distribution of the test set under the target feature type do not meet a preset matching condition;

[0007] Adjust the model training scheme according to the target feature type; the adjusted training scheme is used to control the feature data distribution of the training data under the target feature type and the feature data distribution of the test set under the target feature type in each model training to meet the preset matching conditions;

[0008] The model training is completed using the adjusted training scheme to obtain a candidate model; the candidate model is tested using the test set, and if the test requirements are met, the candidate model is determined as the target model; otherwise, at least one problem that fails to meet the test requirements is determined, the training scheme is adjusted to eliminate the at least one problem, and the process returns to the step of completing the model training using the adjusted training scheme.

[0009] An embodiment of the present application provides a device for automatically diagnosing data, which is applied to an electronic device and includes:

[0010] A quality diagnosis module is used to select images that meet the model training requirements from the first image set to obtain a second image set;

[0011] a classification module, configured to classify the images in the second image set to obtain a training set and a test set; the training set and the test set have at least one common feature type;

[0012] A data analysis module is configured to determine a target feature type; the feature data distribution of the training set under the target feature type and the feature data distribution of the test set under the target feature type do not satisfy a preset matching condition;

[0013] An adaptive optimization module is used to adjust the model training scheme according to the target feature type, and use the adjusted training scheme to complete the model training to obtain a candidate model; the adjusted training scheme is used to control the feature data distribution of the training data under the target feature type for each model training to meet the preset matching conditions with the feature data distribution of the test set under the target feature type;

[0014] The model performance analysis module is used to test the candidate model using the test set. If the test requirements are met, the candidate model is determined to be the target model. Otherwise, at least one problem that fails to meet the test requirements is determined, triggering the adaptive optimization module to adjust the training scheme to eliminate the at least one problem, and continuing to execute the step of completing the model training using the adjusted training scheme.

[0015] An embodiment of the present application further provides an electronic device, comprising: a processor and a machine-readable storage medium;

[0016] The machine-readable storage medium stores machine-executable instructions that can be executed by the processor;

[0017] The processor is used to execute machine-executable instructions to implement the steps of the above method.

[0018] It can be seen from the above technical solutions that in this application, based on the first image set drive, the diagnosis of data in the model training process is fully automated without manual intervention, thereby improving the performance of the final trained model;

[0019] Furthermore, in this embodiment, the training scheme is automatically adjusted based on the characteristic data distribution of the training set and the test set, as well as automatic analysis of potential problems in the model, without the need for human intervention. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0021] Figure 1 A flow chart of the method provided in the embodiment of the present application;

[0022] Figure 2 A flow chart of the method provided in the embodiment of the present application;

[0023] Figure 3 A flowchart for determining images that affect model training provided in an embodiment of the present application;

[0024] Figure 4 Flowchart for implementing step 102 provided in the embodiment of the present application;

[0025] Figure 5 This is a diagram showing the effect of controlling the sampling frequency provided in an embodiment of the present application;

[0026] Figure 6 A flowchart of a method for searching hyperparameters provided in an embodiment of the present application;

[0027] Figure 7 A flowchart for implementing step 104 provided in an embodiment of the present application;

[0028] Figure 8 A diagram of the device structure provided in an embodiment of the present application;

[0029] Figure 9 This is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0031] The terms used in this application are for the purpose of describing particular embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0032] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application, and to make the above-mentioned purposes, features and advantages of the embodiments of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application are further described in detail below with reference to the accompanying drawings.

[0033] In existing model development processes, when problems arise, such as low data quality, substandard calibration, and inconsistent distribution of training data and test data for model testing, experienced professionals are often required to perform professional analysis and data optimization, resulting in a relatively high development threshold. This embodiment abstracts and summarizes the model development process and integrates it into an electronic device that serves as an online model production platform, enabling customers to develop customized algorithms and streamline and tool-enable model training.

[0034] The following describes the embodiments of the present application:

[0035] See also Figure 1 , Figure 1 The flowchart of the method provided in the embodiment of the present application. As an embodiment, the method is applied to the electronic device as the online model production platform as above, where the electronic device is, for example, a PC or other device, which is not specifically limited in this embodiment. The electronic device is configured with an automated diagnostic data tool, which is obtained by tooling the model development process. The electronic device executes the following by running the automated diagnostic data tool: Figure 1 The process shown.

[0036] like Figure 1 As shown, the process may include the following steps:

[0037] Step 101: Select images that meet the model training requirements from the first image set to obtain a second image set; classify the images in the second image set to obtain a training set and a test set; the training set and the test set have at least one common feature type.

[0038] In this embodiment, after the first image set collected on site is calibrated, the execution begins. Figure 1 The automated data diagnosis process shown in FIG. 1 is a flow chart showing how to automatically filter the images in the first image set when executing step 101. The images in the first image set may be automatically filtered to filter out the images that do not meet the model training requirements. Figure 2 One of the implementation methods is shown as an example and will not be described in detail here.

[0039] After obtaining the above-mentioned second image set, the images in the second image set can be classified according to a certain ratio. For example, the images in the second image set are classified according to 7:3, wherein 70% of the images in the second image set are divided into the training set, and the remaining 30% of the images are divided into the test set. For another example, the images in the second image set are classified according to 9:1, wherein 90% of the images in the second image set are divided into the training set, and the remaining 10% of the images are divided into the test set. Optionally, in this embodiment, after the above-mentioned ratio is determined, the training set and the test set can be divided randomly, in chronological order, by image number, etc., which is not specifically limited in this embodiment.

[0040] In this embodiment, the training set and the test set share at least one common feature type. Optionally, in this embodiment, each feature type includes at least one data attribute. This data attribute can include a variety of attributes, such as image-level attributes (e.g., image width, height, brightness, blur), calibration attributes (e.g., object type, character information, object size), and model attributes (e.g., imaging quality, feature distribution), though these are not specifically limited in this embodiment.

[0041] In this embodiment, the training set has a corresponding feature data distribution for each feature type. This feature data distribution is determined based on the feature data corresponding to each data attribute of each training data in the training set under that feature type. Similarly, the test set also has a corresponding feature data distribution for each feature type. The following examples describe how to determine the feature data distribution of the training set under each feature type, and how to determine the feature data distribution of the test set under each feature type. This description is omitted here.

[0042] Step 102 , determining a target feature type; the feature data distribution of the training set under the target feature type and the feature data distribution of the test set under the target feature type do not satisfy a preset matching condition.

[0043] Optionally, in this embodiment, after determining the feature data distribution of the training set and the test set under each feature type, the consistency of the feature data distribution of the training set and the test set under the same feature type will be compared to determine whether the feature data distribution of the training set and the test set under the same feature type meets the preset matching conditions.

[0044] As an example, determining the target feature type may include:

[0045] Step a1: For each feature type, calculate the feature data distribution consistency parameter used to characterize the training set and the test set under the feature type.

[0046] In this embodiment, the more consistent the feature data distribution of the training set and the test set under the same feature type, the better the effect of the candidate model trained below on the test set, and the greater the possibility that the candidate model will be called the target model.

[0047] As an embodiment, the feature data distribution consistency parameter may be a Pearson correlation coefficient, which indicates the consistency between the feature data distribution of the training set under the feature type and the feature data distribution of the test set under the feature type.

[0048] Step a2: If the feature data distribution consistency parameter is less than the set threshold, it is determined that the feature data distribution of the training set and the test set under the feature type does not meet the preset matching condition, and the feature type is the target feature type.

[0049] In this embodiment, still taking the Pearson correlation coefficient as an example of the above-mentioned feature data distribution consistency parameter, the larger the Pearson correlation coefficient, the higher the consistency between the feature data distribution of the training set under this feature type and the feature data distribution of the test set under this feature type, and the more it meets the preset matching conditions, and vice versa.

[0050] Finally, the target feature type mentioned above can be determined through step 102.

[0051] Step 103, adjust the model training scheme according to the target feature type; the adjusted training scheme is used to control the feature data distribution of the training data under the target feature type for each model training to meet the preset matching conditions with the feature data distribution of the test set under the target feature type.

[0052] In this embodiment, step 103 is performed to ensure that the training data used for model training is aligned with the test set, thereby ensuring a reasonable distribution of the training data used for model training. As for how to adjust the model training scheme, examples will be provided below and will not be elaborated on here.

[0053] It can be seen that, through step 103 , this embodiment automatically adjusts the training scheme according to the characteristic data distribution of the training set and the test set without manual intervention.

[0054] Step 104, complete model training using the adjusted training scheme to obtain a candidate model; test the candidate model using the test set, and if the test meets the requirements, determine the candidate model as the target model; otherwise, determine at least one problem that does not meet the above test requirements, adjust the training scheme using the at least one problem to eliminate the at least one problem, and return to the operation of completing model training using the adjusted training scheme.

[0055] Through step 104, this embodiment automatically analyzes potential problems of the model, and guides further optimization of training parameters, automatically adjusts the training plan, and improves model performance.

[0056] So far, completed Figure 1 The process shown.

[0057] pass Figure 1 As can be seen from the process shown, this embodiment is driven by the first image set and fully automates the diagnosis of data during the model training process without manual intervention, making it easy to use.

[0058] Furthermore, in this embodiment, the training scheme is automatically adjusted based on the characteristic data distribution of the training set and the test set, as well as automatic analysis of potential problems in the model, without the need for human intervention.

[0059] The following describes how to select images that meet the model training requirements from the first image set in step 101 above:

[0060] See also Figure 2 , Figure 2 This is a flow chart of the method provided in the embodiment of this application. Figure 2 As shown, the process may include the following steps:

[0061] Step 201 : diagnose whether there are images in the first image set that do not meet the set image requirements, and filter out the images that do not meet the set image requirements from the first image set to obtain a third image set.

[0062] As an example, step 201 can use the trained quality scoring model to diagnose whether there are images in the first image set that do not meet the set image requirements. Optionally, the images that do not meet the set image requirements may include images with defects such as blurry imaging, over-darkness, over-exposure, and excessive occlusion.

[0063] As an example, the aforementioned quality scoring model can be trained based on deep learning. For applications in currently mature services such as facial recognition and license plate recognition, the quality scoring model can be trained using template matching or sample generation combined with pre-adaptation, without requiring real-time updates. For applications in business scenarios where new requirements are constantly emerging and data is constantly being added, the quality scoring model can employ classification or recognition models to directly distinguish valid and invalid data, facilitating online iterative updates. This embodiment is not intended to limit this to specific aspects.

[0064] In this embodiment, images in the first image set that do not meet the set image requirements may be filtered out from the first image set to obtain a third image set. The third image set includes at least images in the first image set that meet the set image requirements. Optionally, in this embodiment, the third image set may also include a quality score for each image.

[0065] Step 202: remove images that affect model training from the third image set to obtain a second image set.

[0066] Optionally, there are many images that affect model training, such as images that meet the following conditions: the calibration result of the image is different from the recognition result of the image through the trained deep model, etc., which is not specifically limited in this embodiment. Figure 3 An example is given to illustrate how to determine images that affect model training, which will not be repeated here.

[0067] So far, completed Figure 2 The process shown.

[0068] pass Figure 2 The process shown implements how to select images that meet the model training requirements from the first image set.

[0069] The following describes the images that affect model training:

[0070] See also Figure 3 , Figure 3 This is a flowchart of determining images that affect model training provided in an embodiment of the present application. Figure 3 As shown, the process may include the following steps:

[0071] Step 301: Obtain a feature vector corresponding to a third image set through a trained first depth model.

[0072] Optionally, in this embodiment, the feature vectors of each image in the third image set in different dimensions such as color, type, etc. can be extracted through the trained first deep model, and then the feature vectors of each image in different dimensions such as color, type, etc. are clustered to finally obtain the feature vectors corresponding to the third image set.

[0073] Step 302 , determining whether the above eigenvector belongs to an open set, if yes, executing step 303 , if no, executing step 304 .

[0074] Optionally, in this embodiment, there are many ways to determine whether the above-mentioned feature vector belongs to an open set, for example, determining whether the above-mentioned feature vector matches a defined feature template. If it does not match, determining that the above-mentioned feature vector belongs to an open set; otherwise, determining that the above-mentioned feature vector does not belong to an open set.

[0075] Step 303: Divide the images in the third image set into K parts. For each L parts of the K parts, use the L parts to train the image diagnosis model, and use the image diagnosis model to predict the images in the remaining KL parts. If it is found that the prediction result of any image is inconsistent with the calibration result of the image, then the image is determined to be an image that affects the model training.

[0076] This step 303 is performed under the premise of an open set. Based on this premise, the operation in step 303 is performed.

[0077] For example, if K is 3 and L is 2, and model training is performed on every two images, three image diagnosis models will eventually be trained. For each image diagnosis model, the image diagnosis model is used to predict each image in the one image that was not used to train the image diagnosis model. If the prediction result for the image is inconsistent with the calibration result, the image is determined to be an image that affects model training.

[0078] In step 303, K-fold verification is performed on the open set to improve the image cleaning accuracy.

[0079] Step 304: For each image in the third image set, use at least two stored domain models to identify the image. If the recognition results do not match, and / or one of the recognition results does not match the calibration result of the image, then the image is determined to be an image that affects model training.

[0080] This step 304 is performed under the premise that the set is not open. Based on this premise, the operation in step 304 is performed.

[0081] In this embodiment, a mismatch in the recognition results may mean that the consistency or similarity of the recognition results does not meet the corresponding set requirements. Similarly, a mismatch between one of the recognition results and the calibration result of the image may also mean that the consistency or similarity of the recognition result and the calibration result of the image does not meet the corresponding set requirements.

[0082] In step 304, under the premise of a non-open set, the stored domain model is directly used to clean images that affect model training, thereby reducing model iteration time.

[0083] So far, completed Figure 3 The process shown.

[0084] Figure 3 The process shown implements different calibration quality diagnosis methods designed for open sets and non-open sets, taking into account both time and accuracy, and ensuring the accuracy of the final second image set.

[0085] The following describes the distribution of feature data of the training set and test set provided in the embodiment of the present application under each feature type:

[0086] As described above, in this embodiment, any feature type includes at least one data attribute. There are many data attributes here, such as image-level attributes such as image width, height, brightness, and blur, calibration attributes such as target type, character information, and target size, and model attributes such as imaging quality and feature distribution. In this embodiment, data attributes are further divided into discrete attributes (such as color, type, etc.) and continuous attributes (such as width, brightness, etc.). Discrete attributes (such as color, type, etc.) and continuous attributes (such as width, brightness, etc.) have different distribution statistics.

[0087] Optionally, in this embodiment, the distribution of feature data under any feature type depends on whether the data attribute in the feature type is a discrete attribute or a continuous attribute. The following takes the training set as an example to describe how to determine the feature data distribution of the training set under the feature type:

[0088] As an example, if a feature type of the training set is represented by a single data attribute, such as color, type, brightness, etc., then when the data attribute is a discrete attribute such as color, type, etc., the feature data (such as frequency, etc., also called discrete value) of each training data in the training set under the data attribute is counted, and the proportion of each discrete value is calculated to obtain the feature data distribution of the training set under the feature type. When the data attribute is a continuous attribute such as width, brightness, etc., optionally, the continuous feature values ​​(also called continuous values) of the training set under the data attribute can be discretized first, and then the feature data distribution of the training set under the feature type can be counted according to the above discrete attribute method.

[0089] Optionally, in this embodiment, there are many ways to discretize continuous values. For example, as an embodiment, for mature applications (such as face recognition), there is already a set of fixed discrete intervals, which can be used to discretize the above-mentioned continuous values. As another embodiment, the maximum and minimum values ​​can be divided equally, or discrete intervals can be determined according to the distribution from small to large, or every x% of data proportion, and then the above-mentioned continuous values ​​can be discretized using the discrete intervals.

[0090] As another embodiment, if a feature type possessed by the training set is represented by a combination of multiple data attributes (such as color + width), the feature data distribution of the training set under the feature type can be determined by combining the various data attributes. For example, if the feature type is obtained by combining two data attributes (such as color + width), if the two data attributes have M and N discrete intervals respectively, then the discrete intervals of the combination of these two data attributes have M*N discrete intervals (i.e., a combination of each discrete interval in the M discrete intervals and each discrete interval in the N discrete intervals). The more data attributes a feature type contains, the more the discrete intervals increase exponentially. Afterwards, the feature data distribution of the training set under the feature type can be statistically calculated based on each discrete interval.

[0091] The above describes how to determine the feature data distribution of the training set under each feature type. The test set is similar and will not be described in detail here.

[0092] The following describes how to adjust the model training scheme according to the target feature type in step 102:

[0093] See also Figure 4 , Figure 4 The step 102 provided in the embodiment of the present application is a flowchart. Figure 4 As shown, the process may include the following steps:

[0094] Step 401 : for each target feature type, if the type of a data attribute in the target feature type is the first type, execute step 402 ; if the type of a data attribute in the target feature type is the second type, execute step 403 .

[0095] In this embodiment, the first type is used to indicate that changes in the feature data corresponding to the data attribute are related to the sampling rate of the training data corresponding to the data attribute from the training set. For example, taking the data attribute of color as an example, the proportion of training data with the color red during each training session is completely determined by the sampling rate of training data with the color red from the training set. Based on this, this embodiment can set the type of data attributes that can be controlled by sampling, such as red, to the first type.

[0096] In this embodiment, the second type is used to indicate adaptive amplification of the characteristic data corresponding to the data attribute. For example, the data attributes are scale, angle, etc. Although the characteristic data corresponding to the data attribute can be controlled by sampling, the effect will be better based on adaptive amplification. Based on this, this embodiment will set the type of data attributes such as scale, angle, etc., which will be better based on adaptive amplification, to the second type.

[0097] Step 402 , based on the fact that the feature data distribution of the training set under the target feature type and the feature data distribution of the test set under the target feature type do not meet the preset matching conditions, adjust the sampling rate of the training data corresponding to the data attribute recorded in the training plan.

[0098] This step 402 is performed under the premise that the type of the data attribute is type 1. When the type of the data attribute is type 1, in this embodiment, the sampling rate recorded in the training plan can be adjusted based on the fact that the feature data distribution of the training set under the target feature type and the feature data distribution of the test set under the target feature type do not meet the preset matching condition.

[0099] Optionally, the situation where the feature data distribution of the training set under the target feature type and the feature data distribution of the test set under the target feature type do not meet the preset matching conditions at least includes: the feature data of the training set and the test set under the data attribute are unbalanced, etc.

[0100] Optionally, in this embodiment, the sampling rate recorded in the training scheme is adjusted based on whether the distribution of feature data for the target feature type in the training set and the distribution of feature data for the target feature type in the test set do not meet a preset matching condition. For example, if the target feature type is represented by a certain data attribute, and the proportion of feature data corresponding to this data attribute in the training set is significantly smaller than the proportion of feature data corresponding to this data attribute in the test set, the sampling rate recorded in the training scheme for training data corresponding to this data attribute is increased, and vice versa. Still using the data attribute red as an example, if the proportion of images with the color red in the training set is significantly smaller than the proportion of images with the color red in the test set, the sampling rate recorded for training data with the color red is increased so that the proportion of images with the color red in each sample is close to the proportion of images with the color red in the test set. Conversely, if the proportion of images with the color red in the training set is significantly larger than the proportion of images with the color red in the test set, the sampling rate recorded for training data with the color red is decreased so that the proportion of images with the color red in each sample is close to the proportion of images with the color red in the test set. Ultimately, through the above adjustments, the difference in data distribution between the training data and the test set for each model training can be reduced, ensuring a more reasonable distribution of training data.

[0101] As an example, when adjusting the sampling rate, a logarithmic function is often used to control the ratio of rare types and reduce the ratio of large samples. The logarithmic function formula is as follows:

[0102] Among them, ori num Indicates the original sampling rate, dst num Indicates the adjusted sampling rate. Figure 5 This example shows the effect of using a logarithmic function to control different sampling rates.

[0103] Step 403: Search for the hyperparameter corresponding to the data attribute by automatic search, add the training data corresponding to the data attribute and the hyperparameter to the training plan as training data for model training and perform model training.

[0104] For example, if the data attributes are scale, angle, etc., the hyperparameters corresponding to the data attributes can be searched automatically, and the training data corresponding to the data attributes can be added to the training plan and multiplied by the hyperparameters as new training data for model training, so that the training data for model training can be aligned with the test set.

[0105] Optionally, in this embodiment, there are many ways to implement the automatic search for the hyperparameters corresponding to the data attributes. Figure 6 One embodiment is shown as an example and will not be described in detail here.

[0106] So far, completed Figure 4 The process shown.

[0107] pass Figure 4 The process shown ultimately realizes how to adjust the model training plan based on the target feature type.

[0108] The following is a summary of the embodiments of this application based on Figure 6 How to implement automatic search to search for the hyperparameters corresponding to the data attributes to describe:

[0109] See also Figure 6 , Figure 6 This is a flow chart of the method for searching hyperparameters provided in the embodiment of the present application. Figure 6 As shown, the process may include the following steps:

[0110] Step 601, for the data attribute, sort the feature data corresponding to the data attribute of each training data in the training set to obtain a training set sequence, and sort the feature data corresponding to the data attribute of each test data in the test set to obtain a test set sequence.

[0111] For example, for a data attribute (such as width), the training set sequence is obtained by sorting the feature data corresponding to the data attribute (for example, if the data attribute is width, the feature data is the width value) from large to small. The principle of the test set sequence is similar and will not be repeated here.

[0112] Step 602: Determine S1 sampling points in the training set sequence, and for each sampling point, perform a specified operation on the feature data associated with the sampling point in the training set sequence to obtain the operation result corresponding to the sampling point, and form the operation results corresponding to each sampling point into a training sampling sequence; and determine S2 sampling points in the test set sequence, and for each sampling point, perform a specified operation on the feature data associated with the sampling point in the test set sequence to obtain the operation result corresponding to the sampling point, and form the operation results corresponding to each sampling point into a test sampling sequence.

[0113] For example, S1 sampling points are selected from the training set sequence according to a set ratio, and the data near each sampling point (for example, 1% of the data, that is, the feature data associated with the sampling point) are subjected to specified operations such as mean and variance to obtain the operation result corresponding to the sampling point.

[0114] Similarly, S2 sampling points can be selected in the test set sequence according to the set ratio, and the data near each sampling point (for example, 1% of the data, that is, the feature data associated with the sampling point) can be subjected to specified operations such as mean and variance to obtain the operation result corresponding to the sampling point.

[0115] In this embodiment, the above S1 and S2 may be equal.

[0116] Step 603 : Based on the distance between the training sampling sequence and the test sampling sequence, a simulated annealing algorithm is used to find the hyperparameter corresponding to the data attribute.

[0117] For example, we can use the simulated annealing algorithm to find the hyperparameters that correspond to the data attributes, with the optimization goal of minimizing the Euclidean distance between the training and test sample sequences. Since the simulated annealing algorithm is a relatively mature one, we will not elaborate on it here.

[0118] Finally passed Figure 6 The process shown here implements an automated search for hyperparameters corresponding to data attributes. For example, if the features corresponding to brightness in the training set are 1, 2, 3, and 4, and the features corresponding to brightness in the test set are 4, 8, 12, and 16, then the hyperparameter can be four times the feature value corresponding to brightness.

[0119] The following describes how to determine at least one problem that causes the test to fail to meet the requirements in step 104:

[0120] See also Figure 7 , Figure 7 The step 104 provided in the embodiment of the present application is implemented as a flow chart. Figure 7 As shown, the process may include the following steps:

[0121] Step 701: For each feature type, determine at least one error interval range by comparing the calibrated feature data distribution of the test set under the feature type with the test feature data distribution of the test set under the feature type tested by the above-mentioned candidate model, and determine whether there is a significant problem interval corresponding to the feature type based on the determined error interval range.

[0122] In this embodiment, the accuracy and error distribution for each feature type are calculated by comparing the distribution of calibrated feature data for the test set under that feature type with the distribution of test feature data for that feature type obtained by testing the candidate model. The accuracy of each feature type is used for subsequent product presentation, while the error distribution of each feature type is used for model error analysis.

[0123] Applied to this embodiment, the error distribution under each feature type is recorded as an error interval range.

[0124] Optionally, in this embodiment, after determining the error interval range, if it is found that the error interval range meets a preset error requirement, then the error interval range is determined to be a significant problem interval. Here, the preset error requirement can be implemented in many ways, such as a percentage greater than a preset threshold, etc., and this embodiment does not specifically limit it. Optionally, in this embodiment, after determining the significant problem interval, the percentage of the significant problem interval is recorded.

[0125] Step 702: For each significant problem interval under each feature type, if the distribution of characteristic data in the significant problem interval meets the preset reasonable requirements, the error cause corresponding to the significant problem interval under the feature type is determined based on the amount of training data in the training set and / or the difference in the distribution of characteristic data between the training set and the error set in the significant problem interval.

[0126] Here, the error set consists of at least the test data with errors in the test set (if the output result of the test data after passing through the candidate model is different from the calibration data of the test data, then the test data has errors).

[0127] Optionally, in this embodiment, each significant problem interval corresponding to each feature type is traversed to first determine whether the significant problem interval is reasonable, such as whether there is feature data close to the critical value (which will cause performance degradation). For significant problem intervals with reasonable distribution, if the feature data distribution of the training set and the error set in this significant problem interval is significantly different, and the training set sample size is less than the threshold, then the error cause corresponding to this significant problem interval for this feature type may be insufficient training set. Otherwise, the error cause is unknown and filtering is performed.

[0128] Step 703: Determine at least one problem that fails to meet the test requirement based on the error cause corresponding to each significant problem interval under each feature type.

[0129] Optionally, in this embodiment, when at least one problem that fails to meet the aforementioned test requirements is determined based on the identified error causes, all unfiltered error causes may be sorted. For example, the error causes may be sorted based on the percentage of significant problem intervals corresponding to the error cause, or based on the difference in feature data distribution between the training set and the error set within the significant problem interval corresponding to the error cause. Here, the greater the error percentage, the greater the difference in feature data distribution, and the more serious the problem.

[0130] Afterwards, the above-mentioned error cause sequence is adjusted. The purpose of the adjustment here is to merge or separate the error causes. For example, some feature types containing multiple data attributes and feature types containing only one data attribute may correspond to the same error in essence, except that the feature type containing multiple data attributes describes the error cause in more detail. In view of this, this embodiment may adjust the error cause sequence when there is an error cause corresponding to the first type of feature type in the error cause sequence, so as to delete the error cause corresponding to the second type of feature type in the error cause sequence. Among them, the first type of feature type only contains one data attribute, and the second type of feature type includes at least two different data attributes and contains the same data attribute as the first type of feature type.

[0131] For example, one of the second-category feature types includes two data attributes of "red + single layer", and the other second-category feature type includes two data attributes of "red + double layer", and there is an error cause corresponding to the data attribute "red" in the error cause sequence. At this time, the error cause corresponding to the second-category feature type, namely "red + single layer", and the error cause corresponding to the second-category feature type "red + double layer" can be deleted from the error cause sequence, so as to replace the error causes corresponding to "red + single layer" and "red + double layer" with the error cause corresponding to "red".

[0132] Finally, at least one error cause is selected from the adjusted error cause sequence as the at least one problem. For example, the top three error problems in the adjusted error cause sequence are output as the at least one problem for subsequent model optimization and adjustment of the model training plan.

[0133] Of course, in this embodiment, if at least one of the above-mentioned questions is not outputted in the end, but the test indicators do not meet the requirements, calibration diagnosis will be automatically performed again or manual review of the calibration accuracy will be required.

[0134] So far, completed Figure 7 The process shown.

[0135] pass Figure 7 The illustrated process implements how to determine at least one issue that caused a test to fail to meet requirements.

[0136] The above describes the method provided in the embodiment of the present application. The following describes the device provided in the embodiment of the present application:

[0137] See also Figure 8 , Figure 8 This is a diagram of the structure of a device provided in an embodiment of the present application. The device is applied to an electronic device and includes:

[0138] A quality diagnosis module is used to select images that meet the model training requirements from the first image set to obtain a second image set;

[0139] a classification module, configured to classify the images in the second image set to obtain a training set and a test set; the training set and the test set have at least one common feature type;

[0140] A data analysis module is configured to determine a target feature type; the feature data distribution of the training set under the target feature type and the feature data distribution of the test set under the target feature type do not satisfy a preset matching condition;

[0141] An adaptive optimization module is used to adjust the model training scheme according to the target feature type, and use the adjusted training scheme to complete the model training to obtain a candidate model; the adjusted training scheme is used to control the feature data distribution of the training data under the target feature type for each model training to meet the preset matching conditions with the feature data distribution of the test set under the target feature type;

[0142] The model performance analysis module is used to test the candidate model using the test set. If the test requirements are met, the candidate model is determined to be the target model. Otherwise, at least one problem that fails to meet the test requirements is determined, triggering the adaptive optimization module to adjust the training scheme to eliminate the at least one problem, and continuing to execute the step of completing the model training using the adjusted training scheme.

[0143] Optionally, in this embodiment, selecting images from the first image set to meet model training requirements includes:

[0144] diagnosing whether there are images in the first image set that do not meet set image requirements, and filtering out images that do not meet the set image requirements from the first image set to obtain a third image set;

[0145] Images that affect model training are removed from the third image set to obtain the second image set.

[0146] Optionally, in this embodiment, the image affecting model training is determined by the following steps:

[0147] Obtain a feature vector corresponding to the third image set through the trained first depth model, and determine whether the feature vector belongs to an open set.

[0148] If yes, the images in the third image set are divided into K parts. For each L parts of the K parts, the image diagnosis model is trained using the L parts, and the images in the remaining KL parts are predicted. If the predicted result of any image is inconsistent with the calibration result of the image, the image is determined to be an image that affects the model training.

[0149] If not, for each image in the third image set, use at least two stored domain models to identify the image. If the recognition results do not match, and / or one of the recognition results does not match the calibration result of the image, then the image is determined to be an image that affects model training.

[0150] Optionally, in this embodiment, determining the target feature type includes:

[0151] For each feature type, calculating a feature data distribution consistency parameter for characterizing the training set and the test set under the feature type;

[0152] If the feature data distribution consistency parameter is less than the set threshold, the feature type is determined to be the target feature type; wherein, the feature data distribution consistency parameter is less than the set threshold, indicating that the feature data distribution of the training set and the test set under this feature type does not meet the preset matching conditions.

[0153] Optionally, in this embodiment, the feature type includes at least one data attribute;

[0154] Adjusting the model training scheme based on the target feature type includes:

[0155] For each data attribute in each target feature type, if the type of the data attribute is a first type, the first type is used to indicate a change in feature data corresponding to the data attribute, and is related to a sampling rate of training data corresponding to the data attribute from a training set, then, based on a situation where the feature data distribution of the training set under the target feature type and the feature data distribution of the test set under the target feature type do not meet a preset matching condition, adjusting the sampling rate recorded in the training plan;

[0156] If the type of the data attribute is the second type, and the second type is used to indicate adaptive amplification of the feature data corresponding to the data attribute, the hyperparameters corresponding to the data attribute are searched through an automatic search method, and the training data corresponding to the data attribute and the hyperparameters are recorded in the training scheme and combined as training data for model training to perform model training.

[0157] Optionally, in this embodiment, searching for hyperparameters corresponding to data attributes by automatic search includes:

[0158] For the data attribute, sorting the feature data corresponding to the data attribute of each training data in the training set to obtain a training set sequence, and sorting the feature data corresponding to the data attribute of each test data in the test set to obtain a test set sequence;

[0159] Determine S1 sampling points in the training set sequence, perform a specified operation on feature data associated with the sampling point in the training set sequence for each sampling point, obtain a calculation result corresponding to the sampling point, and form a training sampling sequence with the calculation results corresponding to each sampling point; and determine S2 sampling points in the test set sequence, perform a specified operation on feature data associated with the sampling point in the test set sequence for each sampling point, obtain a calculation result corresponding to the sampling point, and form a test sampling sequence with the calculation results corresponding to each sampling point;

[0160] According to the distance between the training sampling sequence and the test sampling sequence, a simulated annealing algorithm is used to find hyperparameters corresponding to data attributes.

[0161] Optionally, in this embodiment, determining at least one problem that fails to meet the test requirement includes:

[0162] For each feature type, by comparing the distribution of the calibrated feature data of the test set under the feature type with the distribution of the test feature data of the test set under the feature type tested by the candidate model, at least one error interval range is determined, and based on the determined error interval range, it is determined whether there is a significant problem interval corresponding to the feature type; wherein, when any error interval range meets the preset error requirement, the error interval range is determined to be a significant problem interval;

[0163] For each significant problem interval under each feature type, if the distribution of feature data in the significant problem interval meets preset reasonable requirements, then determine the error cause corresponding to the significant problem interval under the feature type based on the amount of training data in the training set and / or the difference in the distribution of feature data in the significant problem interval between the training set and the error set; the error set is composed of at least test data that have errors in the test set;

[0164] At least one problem that fails to meet the test requirement is determined based on the error cause corresponding to each significant problem interval under each feature type.

[0165] Optionally, in this embodiment, determining at least one problem that fails to meet the test requirement based on the error cause corresponding to each significant problem interval under each feature type includes:

[0166] Sort the determined error causes to obtain an error cause sequence;

[0167] If an error cause corresponding to the first type of feature type exists in the error cause sequence, the error cause sequence is adjusted to delete the error cause corresponding to the second type of feature type in the error cause sequence; the first type of feature type includes only one data attribute, and the second type of feature type includes at least two different data attributes, and one of the data attributes is the same as the data attribute included in the first type of feature type;

[0168] At least one error cause is selected from the adjusted error cause sequence as the at least one problem.

[0169] So far, completed Figure 8 The device structure diagram shown.

[0170] The present application also provides Figure 8 The hardware structure of the device shown. Figure 9 , Figure 9 This is a structural diagram of an electronic device provided in an embodiment of the present application. Figure 9 As shown, the hardware structure may include: a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the method disclosed in the above example of this application.

[0171] Based on the same application concept as the above method, an embodiment of the present application also provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the method disclosed in the above example of the present application can be implemented.

[0172] Exemplarily, the machine-readable storage medium may be any electronic, magnetic, optical, or other physical storage device that may contain or store information, such as executable instructions, data, and the like. For example, the machine-readable storage medium may be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, a storage drive (such as a hard disk drive), a solid-state drive, any type of storage disk (such as a CD, DVD, etc.), or similar storage media, or a combination thereof.

[0173] The systems, devices, modules, or units described in the above embodiments may be implemented by a computer processor or entity, or by a product having certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email transceiver, game console, tablet computer, wearable device, or any combination of these devices.

[0174] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing this application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0175] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the embodiments of the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0176] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0177] Furthermore, these computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0178] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0179] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A method for automated diagnosis of data, characterized in that: The method is applied to an electronic device, comprising: Selecting images that meet model training requirements from the calibrated first image set to obtain a second image set; classifying the images in the second image set to obtain a training set and a test set; the training set and the test set have at least one common feature type; the feature type includes at least one data attribute; Determining a target feature type; the feature data distribution of the training set under the target feature type and the feature data distribution of the test set under the target feature type do not meet a preset matching condition; For each data attribute in each target feature type, if the type of the data attribute is a first type, the first type is used to indicate a change in the feature data corresponding to the data attribute, and is related to the sampling rate of the training data corresponding to the data attribute from the training set, then according to the situation that the feature data distribution of the training set under the target feature type and the feature data distribution of the test set under the target feature type do not meet the preset matching conditions, adjust the sampling rate recorded in the training scheme; if the type of the data attribute is a second type, the second type is used to indicate adaptive amplification of the feature data corresponding to the data attribute, then search for the hyperparameters corresponding to the data attribute by automatic search, record the training data corresponding to the data attribute and the hyperparameters in the training scheme as training data for model training, and perform model training; the adjusted training scheme is used to control the feature data distribution of the training data under the target feature type and the feature data distribution of the test set under the target feature type to meet the preset matching conditions for each model training; The model training is completed using the adjusted training scheme to obtain a candidate model; the candidate model is tested using the test set, and if the test requirements are met, the candidate model is determined as the target model; otherwise, at least one problem that fails to meet the test requirements is determined, the training scheme is adjusted to eliminate the at least one problem, and the process returns to the step of completing the model training using the adjusted training scheme.

2. The method according to claim 1, characterized in that The images selected from the first image set to meet the model training requirements include: diagnosing whether there are images in the first image set that do not meet set image requirements, and filtering out images that do not meet the set image requirements from the first image set to obtain a third image set; Images that affect model training are removed from the third image set to obtain the second image set.

3. The method according to claim 2, characterized in that The images that influence the model training are determined by the following steps: Obtain a feature vector corresponding to the third image set through the trained first depth model, and determine whether the feature vector belongs to an open set. If yes, the images in the third image set are divided into K parts. For each L parts of the K parts, the image diagnosis model is trained using the L parts, and the images in the remaining KL parts are predicted. If the predicted result of any image is inconsistent with the calibration result of the image, the image is determined to be an image that affects the model training. If not, for each image in the third image set, use at least two stored domain models to identify the image. If the recognition results do not match, and / or one of the recognition results does not match the calibration result of the image, then the image is determined to be an image that affects model training.

4. The method according to claim 1, wherein Determining the target feature type includes: For each feature type, calculating a feature data distribution consistency parameter for characterizing the training set and the test set under the feature type; If the feature data distribution consistency parameter is less than the set threshold, the feature type is determined to be the target feature type; wherein, the feature data distribution consistency parameter is less than the set threshold, indicating that the feature data distribution of the training set and the test set under this feature type does not meet the preset matching conditions.

5. The method according to claim 1, wherein The automatic search method for the hyperparameter corresponding to the data attribute includes: For the data attribute, sorting the feature data corresponding to the data attribute of each training data in the training set to obtain a training set sequence, and sorting the feature data corresponding to the data attribute of each test data in the test set to obtain a test set sequence; Determine S1 sampling points in the training set sequence, perform a specified operation on feature data associated with the sampling point in the training set sequence for each sampling point, obtain a calculation result corresponding to the sampling point, and form a training sampling sequence with the calculation results corresponding to each sampling point; and determine S2 sampling points in the test set sequence, perform a specified operation on feature data associated with the sampling point in the test set sequence for each sampling point, obtain a calculation result corresponding to the sampling point, and form a test sampling sequence with the calculation results corresponding to each sampling point; According to the distance between the training sampling sequence and the test sampling sequence, a simulated annealing algorithm is used to find hyperparameters corresponding to data attributes.

6. The method according to claim 1, characterized in that The at least one problem of determining that the test requirement is not met includes: For each feature type, by comparing the distribution of the calibrated feature data of the test set under the feature type with the distribution of the test feature data of the test set under the feature type tested by the candidate model, at least one error interval range is determined, and based on the determined error interval range, it is determined whether there is a significant problem interval corresponding to the feature type; wherein, when any error interval range meets the preset error requirement, the error interval range is determined to be a significant problem interval; For each significant problem interval under each feature type, if the distribution of feature data in the significant problem interval meets preset reasonable requirements, then determine the error cause corresponding to the significant problem interval under the feature type based on the amount of training data in the training set and / or the difference in the distribution of feature data in the significant problem interval between the training set and the error set; the error set is composed of at least test data that have errors in the test set; At least one problem that fails to meet the test requirement is determined based on the error cause corresponding to each significant problem interval under each feature type.

7. The method according to claim 6, characterized in that Determining at least one problem that fails to meet the test requirements based on the error causes corresponding to each significant problem interval under each feature type includes: Sort the determined error causes to obtain an error cause sequence; If an error cause corresponding to the first type of feature type exists in the error cause sequence, the error cause sequence is adjusted to delete the error cause corresponding to the second type of feature type in the error cause sequence; the first type of feature type includes only one data attribute, and the second type of feature type includes at least two different data attributes, and one of the data attributes is the same as the data attribute included in the first type of feature type; At least one error cause is selected from the adjusted error cause sequence as the at least one problem.

8. A device for automated diagnosis of data, characterized in that: The device is applied to electronic equipment including: A quality diagnosis module is used to select images that meet the model training requirements from the calibrated first image set to obtain a second image set; a classification module, configured to classify images in the second image set to obtain a training set and a test set; the training set and the test set have at least one common feature type; the feature type includes at least one data attribute; A data analysis module is configured to determine a target feature type; the feature data distribution of the training set under the target feature type and the feature data distribution of the test set under the target feature type do not satisfy a preset matching condition; An adaptive optimization module is configured to, for each data attribute in each target feature type, if the type of the data attribute is a first type, the first type is used to indicate a change in the feature data corresponding to the data attribute, and is related to a sampling rate of training data corresponding to the data attribute from a training set, then, based on a situation where the feature data distribution of the training set under the target feature type and the feature data distribution of the test set under the target feature type do not meet a preset matching condition, adjust the sampling rate recorded in the training scheme; if the type of the data attribute is a second type, the second type is used to indicate adaptive amplification of the feature data corresponding to the data attribute, then search for hyperparameters corresponding to the data attribute by an automatic search method, record the training data corresponding to the data attribute in the training scheme and combine them with the hyperparameters as training data for model training, and complete the model training using the adjusted training scheme to obtain a candidate model; the adjusted training scheme is configured to control the feature data distribution of the training data under the target feature type and the feature data distribution of the test set under the target feature type to meet a preset matching condition for each model training; The model performance analysis module is used to test the candidate model using the test set. If the test requirements are met, the candidate model is determined to be the target model. Otherwise, at least one problem that fails to meet the test requirements is determined, triggering the adaptive optimization module to adjust the training scheme to eliminate the at least one problem, and continuing to execute the step of completing the model training using the adjusted training scheme.

9. An electronic device, characterized in that: The electronic device includes: a processor and a machine-readable storage medium; The machine-readable storage medium stores machine-executable instructions that can be executed by the processor; The processor is configured to execute machine-executable instructions to implement the method steps of any one of claims 1-7.

Images