Method for evaluating data set quality using task effects, mental health screening system

Abstract

The present invention belongs to the technical field of machine learning, and specifically relates to a method for evaluating the quality of a data set using task effects, a mental health screening system, and a data set quality evaluation system, the method comprising: obtaining a basic feature index BFQI based on a data set; selecting a target task and obtaining a task feature index TFQI and a prediction label based on the task characteristics of the data set; obtaining a task index TQI based on the prediction label; and fusing BFQI, TFQI, and TQI through a weighted function to obtain the task effect DSQI of the data set as an evaluation result of the data set quality. The present invention accurately quantifies the expected effect of the data set on the target task, objectively and comprehensively evaluates the applicability of the data set to tasks such as classification, regression, and clustering, and can not only play a guiding role in the data collection process, but also enable the data sets that are consistent with the target tasks to be merged or the abnormal subjects within the data set to be eliminated, thereby improving the collection efficiency and the effect after being applied to the target task.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine learning, and specifically relates to a method for evaluating the quality of a data set using task effects, a mental health screening system, and a data set quality evaluation system. Background Art

[0002] By combining artificial intelligence methods with brain-computer interface technology, it is possible to construct a large-scale group data set portrait, forming an intelligent brain-computer interface tool that can objectively identify individual task levels. In addition, since EEG, as a physiological signal from the central nervous system, has the advantages of containing a large amount of information and not being easy to disguise, it has become a high-quality data source for emotion recognition. Therefore, group data set portraits can also be used in the fields of mental psychology to screen for emotional disorders, and provide quantitative and objective basis for auxiliary diagnosis of depression, intervention treatment tracking and evaluation, etc.

[0003] However, due to the strong randomness of EEG signals changing with time and the large number of types of complex information contained in the dataset samples, the existing technology usually defines the quality of the dataset simply as the quality of each data in the dataset or a single evaluation dimension. In order to confirm that the current dataset is available, it is necessary to further evaluate whether the data in each category in the dataset has basic consistency in the classification task, and whether the data distribution in the dataset is wide in the regression task. It can be asserted that the dataset with obviously poor values ​​of the above indicators is an unusable dataset, and the model trained on such a dataset will inevitably have poor results. Obviously, the good quality of the collected or preprocessed data is only a prerequisite for the availability of the dataset. It is impossible to observe the dataset comprehensively and objectively, and there is a lack of indicators to measure the expected effect and availability of the dataset on the target task.

[0004] In the related technology, for example, patent CN202410522089.1 discloses a data set quality assessment method, system, electronic device and storage medium, which relies on another target task sample set to return the labels in the sample set for unstructured data and target detection tasks, that is, to detect whether the provided unstructured data contains the "expected detection target" to evaluate the quality of the data set used for the model test set. It does not involve classifying the data, features, basic information, etc. of the structured data set and then testing the quality indicators through the task feature matrix and label vector to obtain the quality of the data set itself when performing a certain target task. Summary of the invention

[0005] The present invention provides a method for evaluating the quality of a data set using task effects, a mental health screening system, and a data set quality evaluation system to quantify the expected effect and usability of a data set on a target task.

[0006] In order to solve the above technical problems, in the first aspect, the present invention provides a method for evaluating the quality of a data set by using task effects, including: obtaining a basic feature indicator BFQI based on the data set; selecting a target task and obtaining a task feature indicator TFQI and a predicted label based on the task characteristics of the data set; obtaining a task indicator TQI based on the predicted label; and fusing the basic feature indicator BFQI, the task feature indicator TFQI, and the task indicator TQI through a weighting function to obtain the task effect DSQI of the data set as an evaluation result of the data set quality.

[0007] Furthermore, when the target task is a supervised learning task, obtaining the task feature indicator includes: recording the data information of the supervised learning task in the data set as the true label; grouping the task features according to the true label, and obtaining the task feature indicator TFQI through hypothesis testing between groups.

[0008] Furthermore, the obtaining of the task indicator includes: performing a hypothesis test on the predicted label and the true label to obtain the task indicator TQI.

[0009] Furthermore, when the supervised learning task is a classification task, the hypothesis test is to calculate the degree of identical distribution between data of different categories; when the supervised learning task is a regression task, the hypothesis test is to calculate the degree of change of statistical characteristics of the data.

[0010] Furthermore, when the target task is an unsupervised learning task, obtaining the task feature index includes: grouping the task features according to the predicted labels, and performing a hypothesis test on the task features between the groups to obtain the task feature index TFQI.

[0011] Furthermore, the obtaining of the task indicator includes: performing a hypothesis test on the predicted label to obtain the task indicator TQI.

[0012] Furthermore, when the unsupervised learning task is a clustering task, the hypothesis test is to calculate the degree of identical distribution between data of different categories.

[0013] Further, the weighting function is configured as task effect DSQI = SQI * BFQI * TQI * TFQI; or task effect DSQI = mean(SQI, BFQI, TQI, TFQI); or task effect DSQI = min(SQI, BFQI, TQI, TFQI); or task effect DSQI = a * SQI + b * BFQI + c * TQI + d * TFQI; wherein a is the data coefficient, b is the basic feature coefficient, c is the task feature coefficient, d is the task coefficient, and SQI is the data indicator.

[0014] Furthermore, the data set includes initial data; the method for obtaining the data indicators includes: obtaining qualified data, that is, screening the initial data above the noise threshold as qualified data and calculating the proportion of qualified data; calculating the distribution statistics of the quality indicators of the initial data to obtain the data indicator SQI; wherein the data indicator SQI = (proportion of qualified data*distribution statistics) / noise threshold; the quality indicators include: at least one of: DC component, offset, power-law distribution coefficient, power frequency intensity, and correlation coefficient.

[0015] Furthermore, the data set includes basic features; the obtaining of basic feature indicators includes: performing hypothesis testing on the basic features to obtain the basic feature indicators BFQI; wherein the basic features include: at least one of the collection information of the data set and the subject information.

[0016] In a second aspect, the present invention provides a mental health screening system, comprising: a processor, running the described method to obtain the evaluation results of the data set for the target task; a human-computer interaction machine connected to the processor, and an evaluation result display area is provided on its operation interface; wherein the evaluation result display area is provided with a data set selection item to select the data set required for mental health screening according to the evaluation results.

[0017] In a third aspect, the present invention provides a data set quality assessment system, comprising: a processor, which runs the steps of the method to obtain an assessment result of the data set for a target task; and a labeling module, which labels the data set according to the assessment result of the target task.

[0018] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the method when executed by a processor.

[0019] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which implements the steps of the method when executed by a processor.

[0020] The beneficial effect of the present invention is that the method for evaluating the quality of a data set using task effects, the mental health screening system, and the data set quality evaluation system of the present invention generate auxiliary labels through training tests, and then fuse the data indicator SQI extracted according to the basic quality of the data, the basic feature indicator BFQI extracted according to the basic features, the task feature indicator TFQI extracted according to the task features, and the task indicator TQI extracted according to the predicted labels together, and weight them to obtain the task effect DSQI of the data set. The present invention accurately quantifies the expected effect of the data set on the target task, objectively and comprehensively evaluates the applicability of the data set to tasks such as classification, regression, and clustering, which can not only play a guiding role in the data collection process, but also merge the data sets that are consistent with the target tasks or eliminate abnormal subjects within the data set, thereby improving the collection efficiency and the effect after application to the target task.

[0021] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0022] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 It is a flowchart of the task effectiveness evaluation method.

[0025] Figure 2 It is a flowchart of extracting task labels.

[0026] Figure 3 It is a flowchart of dimension one for measuring task indicator TQI.

[0027] Figure 4 It is a flowchart of measuring the task indicator TQI in dimension two.

[0028] Figure 5 It is a flowchart of measuring the task indicator TQI in dimension three.

[0029] Figure 6It is a flowchart of the four-dimensional measurement task indicator TQI.

[0030] Figure 7 It is a schematic diagram of the dataset quality assessment system.

[0031] Figure 8 It is the SQI score distribution histogram of the test case.

[0032] Fig. 9 is the age distribution histogram of HC subjects before selection.

[0033] Fig.10 It is the age distribution histogram of MDD subjects before selection.

[0034] Fig.11 is the age distribution histogram of HC subjects after selection.

[0035] Fig.12 It is the age distribution histogram of the MDD subjects after selection.

[0036] Fig.13 It is a graph showing the proportion of task features with significantly different distributions. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] Example 1.

[0039] like Figure 1 As shown, this embodiment 1 provides a method for evaluating the quality of a data set using task effects, including: obtaining a basic feature indicator BFQI based on the data set; selecting a target task and obtaining a task feature indicator TFQI and a predicted label based on the task characteristics of the data set; obtaining a task indicator TQI based on the predicted label; and fusing the basic feature indicator BFQI, the task feature indicator TFQI, and the task indicator TQI through a weighted function to obtain a task effect DSQI of the data set as an evaluation result of the data set quality.

[0040] Datasets usually include structured data and unstructured data. Unstructured data is data without a fixed format or structure, usually in the form of free-form text, multimedia content (such as images, audio and video), emails, social media posts, web pages, etc. If unstructured data is converted into a matrix, it cannot be directly assembled into a tensor form because of the different number of rows and columns. Structured data is data organized according to a predefined data model, with a clearly defined format and structure, usually presented in a table form, and each data field has a clear name and data type. Common structured data includes numbers, text, date and time, price, etc. Structured data can be evaluated by calculating the mean, variance, discreteness and other general evaluation frameworks to evaluate the entire data set. Because the content and characteristics of its samples are similar, the task features can usually be converted into a data matrix and obtain vector labels, which can then form a standardized quality analysis process.

[0041] Optionally, the data set of the present invention is a structured data set, including: initial data, basic features and other information.

[0042] Preferably, the initial data usually contains some low-quality data, which needs to be screened initially. In a specific noise environment, a signal containing noise is collected as a reference for the poor basic quality of the data to set the noise threshold. First, the weighted values ​​of any of the quality indicators such as the DC component DC, offset offset, power law distribution coefficient f-coeff, power frequency intensity PLI, and correlation coefficient R^2 of the test data are calculated as the quantitative index of the data, and then the quantitative index is used to screen the initial data above the noise threshold as the qualified data, and these data are used to calculate the basic characteristic index BFQI, the task characteristic index TFQI, and the task index TQI in the subsequent calculation. Eliminating low-quality data is conducive to improving the accuracy and calculation efficiency of the task effect evaluation. Then, the proportion of qualified data and the distribution statistics of the initial data are calculated (for example, the descriptive distribution statistics such as the mean or standard deviation of all the aforementioned quantitative indicators are calculated), and the data index SQI = (the proportion of qualified data * distribution statistics) / noise threshold is obtained. In particular, if SQI> 1, it is usually truncated and directly taken as 1.

[0043] Optionally, the target tasks include supervised learning tasks, unsupervised learning tasks, etc. In supervised learning tasks, the model is trained on a labeled data set, where the data is paired with corresponding labels. This pairing enables the model to learn and generalize patterns and make accurate predictions on new data. Supervised learning tasks include classification tasks, regression tasks, etc. In unsupervised learning tasks, the model operates on unlabeled data to extract patterns and structures without predefined labels to discover hidden relationships in the data. Unsupervised learning tasks include clustering tasks, anomaly detection tasks, etc.

[0044] Preferably, in the following, the present invention mostly obtains the basic feature index BFQI, the task feature index TFQI, and the task index TQI respectively through the "hypothesis test" method, and through the hypothesis test, judges whether the difference between samples and samples, and between samples and the population is caused by sampling error or essential difference, and then defines the data label, and then further evaluates the possible effect of data application to the task with the help of the data label. Generally, it can be based on the following method: when the target task is a classification task or a clustering task, the hypothesis test method is to calculate the degree of same distribution between different categories of data; when the target task is a regression task, the hypothesis test method is to calculate the degree of change of the statistical characteristics of the data, or the degree of same distribution between the distribution of the data and the uniform distribution.

[0045] As an optional implementation method for obtaining the basic feature index BFQI.

[0046] Optionally, a hypothesis test is performed on the basic characteristics to obtain the basic characteristic index BFQI, and the basic characteristics include the collection information of the data set, the subject information, etc. For example: data source, collection method, collection equipment, collection time, collection location, main experiment number, subject number, subject age, subject gender, subject clinical diagnosis results, etc.

[0047] Specifically, basic features can be divided into quantitative data and qualitative data. For basic features of quantitative data, such as age, BDI scale value, height, ambient temperature, etc., descriptive statistics (calculating mean, median, standard deviation, variance, etc., summarizing sample characteristics), chart analysis (using visualization methods such as histograms and scatter plots to display data distribution and trends), and inferential statistics (including t-test, variance analysis, regression analysis, etc., used to test hypotheses or build models to infer overall characteristics) are usually used for statistical analysis; when the target task is a classification task or a clustering task, the degree of same distribution between different categories of data is tested and the returned P value is recorded as BFQI. When the target task is a regression task, the degree of change of the statistical characteristics of the data (i.e., the divergence of the measured data such as variance, standard deviation, and range) is calculated and recorded as BFQI. For the basic characteristics of qualitative data, such as gender, location, satisfaction rating, depression level, etc., frequency analysis, bar charts or pie charts (using graphical methods to visualize the distribution and proportion of data), chi-square tests, and non-parametric tests are usually used for statistical analysis; when the target task is a classification task or a clustering task, the degree of identical distribution between data of different categories is tested and the returned P value is recorded as BFQI; when the target task is a regression task, the distribution of the data is uniformly analyzed through frequency analysis. If it is uniformly distributed, BFQI=1, if not, BFQI=0; or the probability value of testing that the current data distribution and the uniform distribution belong to the same distribution is BFQI.

[0048] Optionally, after selecting the target task, the data set is input into the model to extract task features (used to assist in obtaining the task feature indicator TFQI) and generate prediction labels (used to assist in obtaining the task indicator TQI). The models include: machine learning models, deep learning models, etc.

[0049] Preferably, see Figure 2 , divide the qualified data in the data set (if no preliminary screening is performed, the data is directly used) into N parts and then input into the model. This division ensures that there are no identical subjects in each part, and the division can be repeated M times, 0 < M ≤ N(N-1); the model is further divided, and the i-th group of data is defined as the training set and put into the training model for preliminary trial training of the task to obtain the task feature Feature_ii, and the j-th group of data is defined as the test set and put into the test model for testing to obtain the task feature Feature_ij. It should be noted that in this embodiment, the task feature is usually a matrix with rows representing sample numbers and columns representing feature dimensions; if the sample is not sliced, the data of a subject is a sample. If slices are made in the time domain, the data of a subject can be divided into multiple samples, that is, the number of matrix rows is the number of subjects * the number of slices. Task features can be regarded as intermediate data from the data obtained through the model to the task label.

[0050] Continuing to explain, the training and testing process of the model extracts task features and generates a predicted label Label_ij. It should be noted that labels can provide context for data that can be understood by machine learning algorithms. These algorithms learn to recognize patterns and make predictions based on labels. Labels (including predicted labels and true labels) are usually vectors that represent sample serial numbers in this embodiment. In classification tasks, labels are usually categories, such as healthy control (HC) subjects and major depressive (MDD) subjects; in regression tasks, a function is obtained through data, and the label is a continuous value predicted by the function for new data; in particular, in unsupervised learning, data usually does not carry real labels, but tasks such as clustering and anomaly detection can obtain quantifiable results returned by each sample data through the model, so the method of this embodiment can also be used to evaluate the effect of the data set.

[0051] Preferably, the task feature is tested by hypothesis to obtain the task feature index TFQI, and the predicted label is tested by hypothesis to obtain the task index TQI. The task feature is intended to abstract the original data, and the task feature index TFQI evaluates the separability and stability of the data; while the predicted label characterizes the expected effect obtained after the data is applied to the target task. This embodiment quantifies this effect by evaluating the difference (separability) between the predicted label and the true label, or the consistency between the predicted labels, or the stability within the predicted label, or the stability of the predicted label after the model is changed.

[0052] As an optional implementation method of obtaining the task feature indicator TFQI.

[0053] When the target task is a supervised learning task, the data information of the supervised learning task in the data set is recorded as the true label; the task features are grouped according to the true label and the type of the target task, and the task feature index TFQI is obtained through the inter-group hypothesis test. Specifically, when the target task is a classification task, the degree of identical distribution between different categories of data is tested and the returned P value is recorded as TFQI; when the target task is a regression task, the degree of change of the statistical characteristics of the data (such as the divergence of the variance, standard deviation, range, etc.) is calculated and recorded as TFQI.

[0054] When the target task is the unsupervised learning task, the data does not contain real labels, so the task features extracted by the model are generalized according to the examples of the predicted labels to be grouped, and the task features are hypothesis tested between the groups to obtain the task feature index TFQI. Specifically, when the target task is a clustering task, the degree of same distribution between different categories of data is calculated and the returned P value is recorded as TFQI.

[0055] The task characteristic index TFQI obtained in this implementation method represents the separability of the data to a certain extent.

[0056] As another optional implementation method of obtaining the task feature indicator TFQI.

[0057] The task features extracted multiple times in the screening model are used to calculate the consistency correlation coefficient CCC of these task features as the task feature index TFQI. The consistency correlation coefficient CCC calculates the stability of the feature values ​​between different data divisions. Therefore, the task feature index TFQI obtained by this implementation method characterizes the stability of the data to a certain extent.

[0058] As an optional implementation method of obtaining the task indicator TQI.

[0059] In general, when the target task is a supervised learning task, the data information of the supervised learning task in the data set is recorded as the true label; the predicted label and the true label are subjected to hypothesis testing to obtain the task indicator TQI. When the target task is the unsupervised learning task, the predicted label is subjected to hypothesis testing to obtain the task indicator TQI. This embodiment provides four dimensions including but not limited to the following to measure the task indicator TQI, and lists the methods of training, testing and hypothesis testing in specific task scenarios.

[0060] Dimension 1: The difference between the predicted label and the true label (separability).

[0061] like Figure 3As shown, the qualified data in the data set (if no preliminary screening is performed, the data is directly used) is divided into N parts and then input into the model. The division is repeated M times, 0 < M ≤ N(N-1). For classification tasks: If M=1, the accuracy ACC of the predicted label and the true label is tested as the TQI; if 1 < M ≤ N(N-1), multiple predicted labels and true labels are tested to obtain the ACC distribution as the distribution of TQI for a single data partition, and the descriptive statistics such as the mean or variance of the distribution are taken as the TQI. For regression tasks: If M=1, the residual value of the predicted label and the true label is tested as the TQI; if 1 < M ≤ N(N-1), multiple predicted labels and true labels are tested to obtain the residual value distribution as the distribution of TQI for a single data partition, and the descriptive statistics such as the mean or variance of the distribution are taken as the TQI.

[0062] Dimension 2: inter-sample stability (consistency between predicted labels).

[0063] like Figure 4 As shown, the qualified data in the data set (if no preliminary screening is performed, the data is directly used) is divided into N parts and then input into the model. The division is repeated M times, 0 < M ≤ N(N-1). In particular, in order to evaluate the stability between samples, in addition to unsupervised learning tasks, supervised learning tasks such as classification and regression tasks also use the algorithm of only performing hypothesis testing on the predicted labels to obtain TQI to improve efficiency. For classification tasks: If M=2, the consistency correlation coefficient CCC between the two predicted labels is tested as the TQI; if 2< M ≤ N(N-1), the CCC distribution of multiple predicted labels is tested as the distribution of TQI for a single data partition, and the descriptive statistics such as the mean or variance of the distribution are taken as the TQI. For regression tasks: If M=2, the correlation coefficient R^2 between the two predicted labels is tested as the TQI; if 2 < M ≤ N(N-1), the R^2 distribution of multiple predicted labels is tested as the distribution of TQI for a single data partition, and the descriptive statistics such as the mean or variance of the distribution are taken as the TQI.

[0064] Dimension 3: Stability within the prediction label.

[0065] like Figure 5As shown, the qualified data in the dataset (if no preliminary screening is performed, the data is directly used) is divided into N parts and then input into the model. The division is repeated M times, where 0 < M ≤ N(N - 1). And, during each division process, the data in the test set within each sample is sliced into P parts from dimensions such as the time domain, frequency domain, or spatial domain, so as to obtain P prediction labels. p prediction labels are selected for hypothesis testing, where 0 < p ≤ P. For classification tasks: test the CCC distribution of the p prediction labels as the distribution of TQI for a single data division; for regression tasks: test the R^2 distribution of the p prediction labels as the distribution of TQI for a single data division. Finally, descriptive statistics such as the mean or variance of the TQI distribution for a single data division are taken as TQI.

[0066] Dimension Four: The stability of prediction labels after changing the model.

[0067] As Figure 6 shown, the qualified data in the dataset (if no preliminary screening is performed, the data is directly used) is divided into N parts and then input into the model. The division is repeated M times, where 0 < M ≤ N(N - 1). And, Q models are successively used for training and testing, so as to obtain Q prediction labels. q prediction labels are selected for hypothesis testing, where 0 < q ≤ Q. The subsequent process follows the method of Dimension Three.

[0068] Optionally, calculate the TQI values according to the selected measurement dimensions respectively, and then weight them according to the attention degree of the attention characteristics of each dimension to obtain the final task index TQI.

[0069] Optionally, the data index SQI ∈ [0, 1]; the basic feature index BFQI ∈ [0, 1]; the task feature index TFQI ∈ [0, 1]; the task index TQI ∈ [0, 1]. The weighted function for calculating the task effect DSQI can include any one of the following functions: DSQI = SQI * BFQI * TQI * TFQI; DSQI = mean(SQI, BFQI, TQI, TFQI); DSQI = min(SQI, BFQI, TQI, TFQI); DSQI = a * SQI + b * BFQI + c * TQI + d * TFQI (where a is the data coefficient, b is the basic feature coefficient, c is the task feature coefficient, and d is the task coefficient, which are set according to the importance degree of the four indexes in the actual scenario, etc.) or other combinations based on the above indexes, then DSQI ∈ [0, 1].

[0070] Example 2.

[0071] Based on Example 1, this Example 2 also provides a mental health screening system, including: a processor, which runs a method for evaluating the quality of a data set using task effects to obtain evaluation results of the data set for one or more target tasks, wherein the evaluation results can be effect categories of "excellent, good, medium, poor", or can be a descriptive record of the task effects; a human-computer interaction machine connected to the processor, wherein an evaluation result display area is provided on its operation interface; wherein a data set selection item is provided on the evaluation result display area, and a doctor can select a data set required for mental health screening according to the evaluation results.

[0072] Through rich emotional situation tasks, the contextualized EEG activity information of individuals is collected, and individual quantitative evaluation can be performed through the multivariate representation similarity method between individuals; through contrastive learning, the core features of quantitatively expressing individual neural responses are extracted to achieve high-performance individual quantitative evaluation. The data set calibrated by this system is used in the fields of mental psychology to screen emotional disorders, and provide quantitative and objective basis for auxiliary diagnosis of depression, intervention treatment tracking and evaluation, etc.

[0073] Example 3.

[0074] See Figure 7 On the basis of Example 1, this Example 3 further provides a data set quality assessment system, including: a processor, which runs a method for assessing data set quality using task effects to obtain the task effect DSQI of the data set for one or more target tasks, so as to obtain the assessment result of the data set for the target task (in the form of score, conclusion label or descriptive record according to the actual scenario); a labeling module, which labels the data set according to the assessment results of different target tasks, that is, calibrates the effect category of the data set for each target task, for simple and rapid classification and integration of various data sets, and for more intuitive and clear selection. The process of data set labeling requires attention to details and precision to ensure that the generated labeled data set accurately represents the real scenario that the data model is expected to encounter.

[0075] Optionally, the effect category can be set to: excellent, good, medium, poor; for example, when the task effect DSQI∈[0.85,1], the effect category is marked as excellent; when the task effect DSQI∈[0.75, 0.84), the effect category is marked as good; when the task effect DSQI∈[0.6, 0.74), the effect category is marked as medium; when the task effect DSQI∈[0, 0.59), the effect category is marked as poor. The effect category and the corresponding DSQI score can also be set according to actual needs.

[0076] Optionally, in supervised learning tasks, if the label of a sample is missing, the data quality of the sample is usually directly judged to be poor; if basic information is missing, the basic feature index BFQI can be discarded and the data index of the sample can be weighted and adjusted based on experience and the coefficient of SQI∈[0,1).

[0077] Test example.

[0078] This test example performs a classification task of "depression binary classification", uses 19 channels of EEG data of 47 healthy control (HC) subjects and 118 subjects with severe depression (MDD) as the test data set, and uses the method of the present invention to evaluate its quality using task effects.

[0079] First, calculate SQI. Set the noise threshold to 90; calculate the DC component DC, offset offset, power law distribution coefficient f-coeff, power frequency intensity PLI, and correlation coefficient R^2 of each test data and weight them to get the SQI score, such as Figure 8 As shown in the distribution histogram of the subjects, the horizontal axis is the SQI score and the vertical axis is the number of subjects. It can be seen that the scores of the 165 subjects are all higher than the noise threshold of 90, and the proportion of qualified data is 100%; then the average of all SQI scores is taken to obtain the distribution statistic value of 96; the calculation results are SQI = 100%* 96 / 90, and the final SQI > 1, so the top value SQI=1.

[0080] Second, calculate BFQI. In order to prevent the model from learning differences in basic information such as gender and age between the two categories, the MDD subjects in the data set are selected based on age information, and only the subject data that matches the age of the HC subject group is used in model training. According to this scheme, the KS test is used for quantitative discrete information such as age, and the rule is set that if p≤threshold, the two types of samples have different distributions, which will interfere with the training accuracy, then BFQI = 0; if p>threshold, then BFQI = 1. For example, Fig. 9 , Fig.10 They are the age distribution histograms of HC and MDD subjects before selection, p = 0.00007, indicating that the two columns of data have different distributions; Fig.11 , Fig.12 The following are the age distribution histograms of the HC and MDD subjects after selection, p = 0.2202, indicating that the two columns of data are identically distributed. When the threshold is set to a significance level higher than 0.05, after selection, p = 0.2202>threshold, BFQI=1 is obtained.

[0081] Third, two methods are used to comprehensively evaluate and calculate TQI.

[0082] Method 1: See Figure 3, the difference (separability) between the predicted label and the true label is measured by dimension one, that is, the data is divided into two groups and then input into the deep learning model, and the accuracy ACC of the predicted label and the true label obtained by 10 data divisions is tested respectively. The results are shown in Table 1; the mean of its distribution is taken as TQI-a=73.5% (or the weighted value of other indicators). The indicator value is relatively high, so it can be expected that the current data set will achieve good classification results when applied to the target task of depression binary classification.

[0083] Table 1.

[0084] Number of divisions 1 2 3 4 5 6 7 8 9 10 Mean Accuracy / % 87.5 90.0 70.0 62.5 70.0 80.0 75.0 70.0 60.0 70.0 73.5

[0085] Method 2: See Figure 5 , the stability of the prediction label is measured by the three-dimensional method, that is, the data is divided into two groups and then input into the deep learning model. In the model, the first group of data is used as the training set, and the second group of data is used as the test set and is divided into the first part of the test set and the second part of the test set. The predicted label Label_121 and the predicted label Label_122 are tested respectively; the current classification task uses CCC as the evaluation index, and the data division is repeated 10 times. The CCC distribution of the two predicted labels is tested as the distribution of TQI for a single data division; the distribution obtained 10 times is averaged, and the results are shown in Table 2. The mean values ​​of the HC and MDD test data obtained by this process are 0.9180 and 0.9054, respectively. The average value is TQI-b=0.9117, which is a high indicator value. Therefore, the data is evaluated to be stable in the time dimension, and it is expected that the current data set will achieve good classification results when applied to the target task of depression binary classification.

[0086] Table 2.

[0087] Number of divisions HC-CCC MDD-CCC average value 1 0.9367 0.9243 0.9305 2 0.9186 0.8102 0.8644 3 0.9124 0.8715 0.8920 4 0.9040 0.9230 0.9135 5 0.9599 0.9802 0.9701 6 0.9789 0.9596 0.9692 7 0.8929 0.8810 0.8869 8 0.8936 0.8100 0.8518 9 0.9688 0.9107 0.9398 10 0.8146 0.9840 0.8993 Mean 0.9180 0.9054 0.9117

[0088] Finally, separability is more important, so method 1 is assigned a weighting coefficient of 0.9 and method 2 is assigned a weighting coefficient of 0.1, and the final TQI is obtained as follows: 0.9*(TQI-a)+ 0.1*(TQI-b)=0.9*73.5%+0.1*0.9117=75.27%.

[0089] Fourth, calculate TFQI. In this experimental example (i.e., 10 data divisions, using a specific model), the task feature matrix is ​​a three-dimensional matrix of 10*96*number of samples, where the first dimension 10 of the task feature matrix represents that 10 data divisions were made, the second dimension 96 represents that the fixed feature dimension extracted by the model used is 96, and the third dimension number of samples is because each sample will extract the characteristic value of each dimension in each data division. The matrix is ​​divided into positive task features and negative task features according to positive and negative categories, and the corresponding dimensions between the positive and negative task features are statistically tested. Since the HC subjects and MDD subjects are not paired one by one in terms of numbering, it is possible to consider using KS to test whether the two types of features are distributed in the same way, h = 1, indicating that the two columns of data are distributed differently, and p = 1.5205e-11 indicates significance. Using this test method for all 96-dimensional features, the following is obtained: Fig.13 As shown in the graph of the proportion of task features with significantly different distributions (10 rows * 96 columns), the proportion of feature color blocks with significantly different distributions is 90.83%, that is, TFQI=90.83%.

[0090] To summarize, for this test example, DSQI = SQI * BFQI * TQI * TFQI = 1 * 1 *75.27%* 90.83% = 0.6837.

[0091] Example 4.

[0092] On the basis of Embodiment 1, this Embodiment 4 further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method are implemented.

[0093] Example 5.

[0094] On the basis of Embodiment 1, this Embodiment 5 further provides a computer program product, including a computer program, which implements the steps of the method when the computer program is executed by a processor.

[0095] The method for evaluating the quality of a data set using task effects, the mental health screening system, and the data set quality evaluation system of the present invention generate labels for auxiliary classification through training and testing, and fuse the data indicator SQI extracted according to the basic quality of the data, the basic feature indicator BFQI extracted according to the basic feature, the task feature indicator TFQI extracted according to the task feature, and the task indicator TQI extracted according to the predicted label together, and weight them to obtain the task effect DSQI of the data set. The present invention can be used for both supervised learning tasks and unsupervised learning tasks, accurately quantifies the expected effect of the data set on the target task, and objectively and comprehensively evaluates the applicability of the data set to tasks such as classification, regression, and clustering. It can not only play a guiding role in the data collection process, but also merge data sets that are consistent with the target task or eliminate abnormal subjects within the data set, thereby improving the collection efficiency and the effect of a single sample after being applied to the target task.

[0096] In the description of the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0097] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0098] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0099] Based on the above ideal embodiments of the present invention, the relevant staff can make various changes and modifications without departing from the technical concept of the present invention through the above description. The technical scope of the present invention is not limited to the contents of the specification, and its technical scope must be determined according to the scope of the claims.

Claims

1. A mental health screening system, characterized in that: include: A processor, running a method for evaluating the quality of a data set using a task effect to obtain an evaluation result of the data set for a target task; the data set is configured as EEG data; A human-computer interaction machine connected to the processor, wherein an evaluation result display area is provided on its operation interface; in The evaluation result display area is provided with a data set selection item to select the data set required for mental health screening according to the evaluation result; The method for evaluating the quality of a data set using task effects includes: Obtain the basic feature index BFQI based on the data set; Select the target task and obtain the task feature index TFQI and the predicted label according to the task characteristics of the data set; the task feature index TFQI represents the separability and stability of the data, and the predicted label represents the expected effect of the data applied to the target task; Obtaining a task indicator TQI based on the predicted label; wherein the task indicator TQI represents the difference between the predicted label and the true label, or the consistency between the predicted labels, or the stability within the predicted label, or the stability of the predicted label after the model is replaced; and The basic feature index BFQI, task feature index TFQI and task index TQI are fused through a weighted function to obtain the task effect DSQI of the data set as the evaluation result of the data set quality.

2. The mental health screening system according to claim 1, characterized in that: When the target task is a supervised learning task, the task characteristic indicators to be acquired include: Recording the data information of the supervised learning task in the data set as the true label; The task features are grouped according to the true labels, and the task feature index TFQI is obtained through hypothesis testing between groups.

3. The mental health screening system according to claim 2, characterized in that: The obtaining of the task indicator includes: performing a hypothesis test on the predicted label and the true label to obtain the task indicator TQI.

4. The mental health screening system according to claim 2 or 3, characterized in that: When the supervised learning task is a classification task, the hypothesis test is to calculate the degree of identical distribution between data of different categories; When the supervised learning task is a regression task, the hypothesis test is to calculate the degree of change of the statistical characteristics of the data.

5. The mental health screening system according to claim 1, characterized in that: When the target task is an unsupervised learning task, obtaining the task feature index includes: grouping the task features according to the predicted labels, and performing a hypothesis test on the task features between the groups to obtain the task feature index TFQI.

6. The mental health screening system according to claim 5, characterized in that: The acquiring of the task indicator includes: performing a hypothesis test on the predicted label to obtain the task indicator TQI.

7. The mental health screening system according to claim 5 or 6, characterized in that: When the unsupervised learning task is a clustering task, the hypothesis test is to calculate the degree of identical distribution between data of different categories.

8. The mental health screening system according to claim 1, characterized in that: The weighting function is configured as Mission effectiveness DSQI = SQI * BFQI * TQI * TFQI; or Task effect DSQI = mean(SQI, BFQI, TQI, TFQI); or Task effect DSQI = min(SQI, BFQI, TQI, TFQI); or Task effect DSQI = a * SQI + b * BFQI + c * TQI + d * TFQI; where a is the data coefficient, b is the basic characteristic coefficient, c is the task characteristic coefficient, d is the task coefficient, and SQI is the data indicator.

9. The mental health screening system according to claim 8, characterized in that: The data set includes initial data; Methods for obtaining the data indicators include: Obtain qualified data, that is, select initial data that is higher than the noise threshold as qualified data and calculate the proportion of qualified data; Calculate the distribution statistics of the quality index of the initial data to obtain the data index SQI; The data indicator SQI = (percentage of qualified data * distribution statistics) / noise threshold; The quality index includes at least one of a DC component, an offset, a power law distribution coefficient, a power frequency intensity, and a correlation coefficient.

10. The mental health screening system according to claim 1, characterized in that: The data set includes basic features; The obtaining of the basic characteristic index comprises: performing a hypothesis test on the basic characteristic to obtain the basic characteristic index BFQI; wherein The basic features include: at least one of data set collection information and subject information.

Images