Data detection method and device and storage medium
By entering the subsatisfaction of the satisfaction questionnaire data into the classification model, combined with preset grouping rules, the problem of abnormal detection in satisfaction sequenced type data is solved, and the reliability of effective abnormal detection and data analysis of satisfaction data is realized. promote.
Patent Information
- Application Number
- CN202311030349.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-15
- Publication Date
- 2025-02-21
AI Technical Summary
The prior art cannot effectively detect abnormal data in sequencing type data such as satisfaction, and abnormal detection cannot be performed through the detection of numerical data.
By obtaining the target questionnaire data, input the target subsatisfaction into the target classification model, obtain the classification results, and determine the first and second groups according to the preset grouping rules, thereby determining the abnormal detection results of the questionnaire data.
The abnormal detection of satisfaction-sequential type data is realized, which improves the reliability of data analysis and can effectively eliminate abnormal data.
Smart Images
Figure CN119494680A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of data processing technology, and in particular to a data detection method, device and storage medium. Background Art
[0002] The questionnaire structure based on satisfaction includes sub-satisfaction corresponding to the sub-items and the overall total satisfaction. For example, for a satisfaction survey on a purchased service, first set up the total satisfaction item, and then set up the sub-survey items corresponding to the sub-satisfaction, such as the logistics packaging satisfaction sub-survey item, the logistics timeliness satisfaction sub-survey item, etc. Among them, the sub-satisfaction is an important factor in the total satisfaction.
[0003] In the related art, anomaly detection is usually performed on numerical data, but for ordinal data such as satisfaction, anomaly detection cannot be performed using the detection method of numerical data. Summary of the invention
[0004] In order to overcome the problems existing in the related art, the present disclosure provides a data detection method, device and storage medium.
[0005] According to a first aspect of an embodiment of the present disclosure, there is provided a data detection method, including:
[0006] Obtaining target questionnaire data, wherein the target questionnaire data includes target sub-satisfaction and target total satisfaction;
[0007] Inputting the target sub-satisfaction of the target questionnaire data into the target classification model to obtain the target classification result corresponding to the target questionnaire data, wherein the target classification result is used to characterize the classification probability corresponding to the predicted total satisfaction of the target questionnaire data;
[0008] According to a preset grouping rule, determining a first group corresponding to the target classification result and a second group corresponding to the target total satisfaction;
[0009] An abnormality detection result of the target questionnaire data is determined according to the first grouping and the second grouping.
[0010] Optionally, the number of the target questionnaire data is multiple, and determining the first group corresponding to the target classification result according to a preset grouping rule includes:
[0011] Determine the number of target groups according to multiple target classification results corresponding to multiple target questionnaire data;
[0012] Sorting the multiple target classification results according to a preset order to obtain a target sequence classification result;
[0013] The target sequential classification results are divided based on the number of target groups, and a first group corresponding to each target classification result is determined.
[0014] Optionally, determining the number of target groups according to multiple target classification results corresponding to multiple target questionnaire data includes:
[0015] Determine the target index value of the target classification model according to the target classification result;
[0016] Multiply the maximum value of the value range of the target total satisfaction by the reciprocal of the target indicator value to obtain the initial number of groups;
[0017] The initial number of groups is rounded up to obtain the target number of groups.
[0018] Optionally, determining the second group corresponding to the target total satisfaction level according to a preset grouping rule includes:
[0019] According to the target total satisfaction and the preset correspondence between the total satisfaction and the grouping, a second grouping corresponding to the target total satisfaction is determined.
[0020] Optionally, the first group and the second group have corresponding group serial numbers, and determining the abnormality detection result of the target questionnaire data according to the first group and the second group includes:
[0021] Dividing the group number corresponding to the first group by the group number corresponding to the second group to obtain a target score corresponding to the target questionnaire data;
[0022] The abnormality detection result of the target questionnaire data is determined according to the magnitude relationship between the target score and the preset score threshold.
[0023] Optionally, the preset score threshold is determined in the following manner:
[0024] Determine the target index value of the target classification model according to the target classification result;
[0025] Multiply the maximum value of the target total satisfaction value range by the inverse of the target indicator value to obtain the upper limit threshold of the preset score threshold, and use the inverse of the maximum value of the target total satisfaction value range as the lower limit threshold of the preset score threshold.
[0026] Optionally, determining the abnormality detection result of the target questionnaire data according to the magnitude relationship between the target score and a preset score threshold includes:
[0027] When the target score is equal to the upper threshold or the lower threshold of the preset score threshold, an abnormality detection result is determined that characterizes the target questionnaire data as abnormal questionnaire data.
[0028] Optionally, the target classification model is constructed in the following manner:
[0029] Determining an initial classification model and sample questionnaire data, wherein the sample questionnaire data includes sample sub-satisfaction and sample total satisfaction;
[0030] Determine a sample label of the sample questionnaire data according to a preset label threshold and a sample total satisfaction level of the sample questionnaire data;
[0031] The initial classification model is trained based on the sample labels and the sample sub-satisfaction of the sample questionnaire data to obtain the target classification model after model training.
[0032] According to a second aspect of an embodiment of the present disclosure, a data detection device is provided, the device comprising:
[0033] An acquisition module is configured to acquire target questionnaire data, wherein the target questionnaire data includes target sub-satisfaction and target total satisfaction;
[0034] An input module is configured to input the target sub-satisfaction of the target questionnaire data into a target classification model to obtain a target classification result corresponding to the target questionnaire data, wherein the target classification result is used to characterize a classification probability corresponding to the predicted total satisfaction of the target questionnaire data;
[0035] A grouping module, configured to determine a first group corresponding to the target classification result and a second group corresponding to the target total satisfaction according to a preset grouping rule;
[0036] A determination module is configured to determine an abnormality detection result of the target questionnaire data according to the first grouping and the second grouping.
[0037] According to a third aspect of an embodiment of the present disclosure, there is provided a data detection device, including:
[0038] processor;
[0039] a memory for storing processor-executable instructions;
[0040] Wherein, the processor is configured to:
[0041] Obtaining target questionnaire data, wherein the target questionnaire data includes target sub-satisfaction and target total satisfaction;
[0042] Inputting the target sub-satisfaction of the target questionnaire data into the target classification model to obtain the target classification result corresponding to the target questionnaire data, wherein the target classification result is used to characterize the classification probability corresponding to the predicted total satisfaction of the target questionnaire data;
[0043] According to a preset grouping rule, determining a first group corresponding to the target classification result and a second group corresponding to the target total satisfaction;
[0044] An abnormality detection result of the target questionnaire data is determined according to the first grouping and the second grouping.
[0045] According to a fourth aspect of an embodiment of the present disclosure, there is provided a computer-readable storage medium on which computer program instructions are stored, and when the program instructions are executed by a processor, the steps of any one of the methods described in the first aspect are implemented.
[0046] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:
[0047] First, the target questionnaire data is obtained, and then the target sub-satisfaction of the target questionnaire data is input into the target classification model to obtain the target classification result corresponding to the target questionnaire data, and then according to the preset grouping rules, the first group corresponding to the target classification result and the second group corresponding to the target total satisfaction are determined, and finally, according to the first group and the second group, the abnormal detection result of the target questionnaire data is determined. Using the above method, the target sub-satisfaction is classified and predicted by the target classification model to obtain the target classification result, and then the abnormal detection result of the target questionnaire data is determined according to the group where the target classification result is located and the group where the target total satisfaction is located, thereby realizing abnormal detection for the ordered type data such as satisfaction. In addition, in the subsequent data statistical analysis process, abnormal data can be eliminated according to the abnormal detection results, thereby improving the reliability of data analysis.
[0048] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0050] Figure 1 is a flow chart of a data detection method according to an exemplary embodiment;
[0051] Figure 2 is a flowchart showing a questionnaire data processing according to an exemplary embodiment;
[0052] Figure 3is a block diagram of a data detection device according to an exemplary embodiment;
[0053] Figure 4 It is a block diagram of a data detection device according to an exemplary embodiment. DETAILED DESCRIPTION
[0054] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.
[0055] It should be noted that all actions of acquiring signals, information or data in the present disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the device is located and with the authorization given by the owner of the corresponding device.
[0056] It is worth noting that numerical data is an observation value measured on a digital scale, and the result is expressed as a specific numerical value, such as age, test scores, temperature, etc. Common anomaly detection methods for numerical data are the 3σ principle, box plot method, threshold determination method, etc. For details, please refer to the relevant technology, and this disclosure will not go into details here. Taking age data for anomaly detection as an example, the normal threshold is set to 0-120 years old. If the age is 200 years old, it can be determined as abnormal data according to the threshold determination method.
[0057] Satisfaction is ordinal data, which is measured by ordinal scales and is expressed as categories. It can be sorted, but can only be compared in size, and cannot perform mathematical operations. For example, both the total survey items and sub-survey items include satisfaction options such as "unsatisfied, generally satisfied, satisfied, relatively satisfied, and very satisfied", or the satisfaction options can also be numerical, such as 1-10 points. For data anomaly detection of satisfaction survey questionnaires, only abnormal data such as null values can be simply judged. However, if the sub-survey items are all 1 point or "unsatisfied", but the total survey item is 10 points or "very satisfied", it is considered abnormal data from the logic of the satisfaction survey. However, the anomaly detection method in the relevant technology cannot be applied to this scenario, that is, it cannot detect abnormal data in the ordinal data.
[0058] In view of this, the present disclosure provides a data detection method, device and storage medium to solve the above technical problems.
[0059] Figure 1 is a flow chart of a data detection method according to an exemplary embodiment. Figure 1 As shown, the following steps are included.
[0060] In step S11, target questionnaire data is acquired.
[0061] Among them, the target questionnaire data includes target sub-satisfaction and target total satisfaction.
[0062] For example, the structure of the satisfaction survey questionnaire is a total score structure, and the questionnaire content includes survey items for overall satisfaction (target total satisfaction) and sub-survey items for target sub-satisfaction. The target sub-satisfaction is an important factor in constituting the target total satisfaction. The satisfaction survey questionnaire can be designed according to the needs, such as a satisfaction survey questionnaire for purchasing services, a satisfaction survey questionnaire for gaming experience, etc., and the present disclosure does not limit this. Taking the satisfaction survey questionnaire for purchasing services as an example, the sub-satisfaction can be other business process satisfaction that affects the overall satisfaction, such as logistics timeliness satisfaction, logistics packaging satisfaction, etc., and the present disclosure does not limit this.
[0063] It should be noted that the sampling group of the questionnaire can be set according to the needs. For example, for the satisfaction questionnaire of purchasing services, the sampling group can be customers who have purchased within a certain period of time, and the target sampling customers can be determined by random sampling, stratified sampling, systematic sampling and other sampling methods. The questionnaire can be distributed through SMS, Internet and other methods, and the present disclosure does not limit this. It is worth noting that the collection of satisfaction questionnaires in the embodiments of the present disclosure is carried out with the authorization of the user, for example, through the online questionnaire. After the user fills in the questionnaire, click Submit to indicate that he agrees to authorize the questionnaire distributor to obtain the questionnaire data.
[0064] In a possible way, the collected satisfaction survey questionnaire data can be preliminarily cleaned, for example, by setting strong cleaning rules, such as incomplete questionnaires, etc. Rules are matched for each questionnaire, and if a questionnaire meets the set strong cleaning rules, the questionnaire is deleted. This can preliminarily screen out some simple abnormal data, reduce the workload of subsequent data detection, and improve the efficiency of data detection.
[0065] In step S12, the target sub-satisfaction of the target questionnaire data is input into the target classification model to obtain the target classification result corresponding to the target questionnaire data.
[0066] Among them, the target classification result is used to characterize the classification probability corresponding to the predicted total satisfaction of the target questionnaire data.
[0067] It should be noted that when the target sub-satisfaction is numerical data, it can be directly input into the target classification model. When the sub-satisfaction is text data such as "dissatisfied, generally satisfied, satisfied, relatively satisfied, very satisfied", if the target classification model supports the input of text data, it can be directly input. If the target classification model supports numerical data, it can be converted into numerical data for input, for example, "dissatisfied" is 1, and "very satisfied" is 10. The present disclosure does not impose any restrictions on this.
[0068] In a possible manner, the target classification model is constructed in the following manner: determining an initial classification model and sample questionnaire data, the sample questionnaire data including sample sub-satisfaction and sample total satisfaction; determining a sample label of the sample questionnaire data according to a preset label threshold and the sample total satisfaction of the sample questionnaire data; training the initial classification model based on the sample label and the sample sub-satisfaction of the sample questionnaire data to obtain a target classification model after model training.
[0069] For example, the initial classification model can be a logistic model, a RF (random forests) classification model, a GBDT (Gradient Boosting Decision Tree) classification model, an XGB (XGBoost) classification model, a deep learning classification model, etc., which can be determined according to the needs, and the present disclosure does not limit this. Set a preset label threshold γ, and set the satisfaction level greater than or equal to γ to 1, otherwise it is 0. For example, the target total satisfaction includes "unsatisfied, generally satisfied, satisfied, relatively satisfied, very satisfied", set the satisfaction level greater than or equal to "satisfied" to 1, otherwise it is 0, or the target total satisfaction is 1-10 points, set the satisfaction level greater than or equal to 6 points to 1, otherwise it is 0, and so on, the present disclosure does not limit this.
[0070] For example, according to the label threshold and the total sample satisfaction of each sample questionnaire data, the sample label corresponding to each sample questionnaire data is determined to obtain label data. Then, the model is trained according to the sample sub-satisfaction and label data of the sample questionnaire data to obtain the final target classification model. For details, please refer to the training process of the binary classification model in the relevant technology, which will not be repeated in this disclosure. In addition, it is worth noting that the sample questionnaire data can be the same as the target questionnaire data, or different from the target questionnaire data, but the sample questionnaire data and the target questionnaire data are both questionnaire data obtained for the same questionnaire.
[0071] In a possible manner, the target classification result is used to characterize the classification probability corresponding to the predicted total satisfaction of the target questionnaire data. For example, the target classification result can be the probability that the classification corresponding to the predicted total satisfaction of the target questionnaire data is 1 (for example, the predicted total satisfaction is greater than or equal to "satisfied").
[0072] In step S13, according to a preset grouping rule, a first group corresponding to the target classification result and a second group corresponding to the target total satisfaction are determined.
[0073] In a possible manner, there are multiple target questionnaire data, and determining the first group corresponding to the target classification result according to a preset grouping rule may include: determining the number of target groups according to multiple target classification results corresponding to the multiple target questionnaire data; sorting the multiple target classification results according to a preset order to obtain target sequence classification results; dividing the target sequence classification results based on the number of target groups, and determining the first group corresponding to each target classification result.
[0074] For example, taking 100 target questionnaire data as an example, multiple target classification results can be arranged in order from small to large or from large to small to obtain target order classification results. Then, the target order classification results are divided according to the number of target groups, which can be divided equally or unequally. For example, if the number of target groups is 5, 1-20 is the first group, 20-40 is the second group, and so on (in order from small to large). If it is in order from large to small, 1-20 is the fifth group. Thus, the group to which each target classification result belongs is determined as the first group corresponding to the target classification result.
[0075] In a possible manner, the number of target groups can be set according to demand, for example, according to the total number of groups corresponding to the total satisfaction, or according to multiple target classification results corresponding to multiple target questionnaire data, and the present disclosure does not limit this. Among them, determining the number of target groups according to multiple target classification results corresponding to multiple target questionnaire data can include: determining the target index value of the target classification model according to the target classification results; multiplying the maximum value of the value range of the target total satisfaction by the inverse of the target index value to obtain the initial number of groups; rounding up the initial number of groups to obtain the target number of groups.
[0076] For example, the target index value may be an ACU (Area Under Curve) index, or other performance index for evaluating a classification model, which is not limited in the present disclosure. The calculation method of the ACU index may refer to related technologies, which will not be described in detail in the present disclosure. The target number of groups may be determined according to the following calculation formula:
[0077] Grp_new=ceil[(1 / AUC)×grp_num]
[0078] Among them, Grp_new represents the number of target groups, AUC represents the AUC index, grp_num represents the maximum value of the value range of the target total satisfaction, and ceil represents the upward rounding function.
[0079] In a possible manner, determining the second group corresponding to the target total satisfaction according to a preset grouping rule may include: determining the second group corresponding to the target total satisfaction according to the target total satisfaction and a preset correspondence between the total satisfaction and the grouping.
[0080] For example, the value range of total satisfaction can be grouped to obtain a preset correspondence between total satisfaction and grouping, such as 1 point for Group 1, 2 points for Group 2, or "unsatisfied" for Group 1, and so on.
[0081] It should be noted that the maximum value of the target total satisfaction range can also be the number of groups of total satisfaction. For example, the value range of total satisfaction is 1-100 points, and the groups of total satisfaction can be divided into 100 groups or 10 groups, that is, grp_num can be the number of groups of total satisfaction, and the present disclosure does not impose any restrictions on this.
[0082] In step S14, an abnormality detection result of the target questionnaire data is determined according to the first grouping and the second grouping.
[0083] In a possible manner, the first group and the second group correspond to group numbers, and determining the abnormality detection result of the target questionnaire data based on the first group and the second group can include: dividing the group number corresponding to the first group by the group number corresponding to the second group to obtain the target score corresponding to the target questionnaire data; determining the abnormality detection result of the target questionnaire data based on the relationship between the target score and the preset score threshold.
[0084] For example, the group number corresponding to the first group is divided by the group number corresponding to the second group to obtain a target score, and then the abnormality detection result of the target questionnaire data is determined according to the relationship between the target score and a preset score threshold.
[0085] In a possible manner, the preset score threshold is determined in the following manner: determine the target indicator value of the target classification model based on the target classification result; multiply the maximum value of the target total satisfaction value range by the inverse of the target indicator value to obtain the upper limit threshold of the preset score threshold, and use the inverse of the maximum value of the target total satisfaction value range as the lower limit threshold of the preset score threshold.
[0086] For example, the preset score threshold can be determined according to the following calculation formula:
[0087] Score-max = (1 / AUC) × grp_num
[0088] Score-min = 1 / ×grp_num
[0089] Among them, Score-max represents the upper limit threshold of the preset score threshold, and Score-min represents the lower limit threshold of the preset score threshold.
[0090] It should be noted that the distance measured by Score-max is that a certain target questionnaire data is simultaneously divided into the smallest group of total satisfaction (the second group is the smallest serial number) and the largest group of model prediction grouping (the first group is the largest serial number), and the distance measured by Score-min is that a certain target questionnaire data is simultaneously divided into the largest group of total satisfaction (the second group is the smallest serial number) and the smallest group of model prediction grouping (the first group is the smallest serial number).
[0091] In a possible manner, determining the abnormality detection result of the target questionnaire data based on the size relationship between the target score and the preset score threshold may include: when the target score is equal to the upper threshold or lower threshold of the preset score threshold, determining the abnormality detection result that characterizes the target questionnaire data as abnormal questionnaire data.
[0092] For example, when the target score is equal to the upper threshold or lower threshold of the preset score threshold, the classification result corresponding to the sub-satisfaction of the target questionnaire data is in the smallest group, while the total satisfaction is in the largest group, or the classification result corresponding to the sub-satisfaction of the target questionnaire data is in the largest group, while the total satisfaction is in the smallest group. In this case, it is determined that the target questionnaire data is an abnormal detection result of abnormal questionnaire data. Generally speaking, this situation occurs when the questionnaire data is a sub-survey item with a score of 1 or "unsatisfied", but the total survey item is 10 points or "very satisfied". This kind of questionnaire is generally filled out by the user at random and has no statistical analysis significance, so it can be identified as abnormal data. When the target score is not equal to the upper threshold of the preset score threshold and is not equal to the lower threshold, it is determined that the target questionnaire data is an abnormal detection result of non-abnormal questionnaire data.
[0093] In other possible implementations, determining the abnormality detection result of the target questionnaire data based on the first grouping and the second grouping can include: dividing the group number corresponding to the first group by the first total number of groups to obtain a first ratio, and dividing the group number corresponding to the second group by the second total number of groups to obtain a second ratio; dividing the first ratio by the second ratio to obtain a questionnaire score corresponding to the target questionnaire data; and determining the abnormality detection result of the target questionnaire data based on the relationship between the questionnaire score and a preset threshold.
[0094] Among them, according to the relationship between the questionnaire score and the preset threshold, the abnormal detection result of the target questionnaire data is determined, which can be: when the questionnaire score is greater than or equal to the upper threshold of the preset threshold or less than or equal to the lower threshold, the abnormal detection result representing the target questionnaire data as abnormal questionnaire data is determined. Or, when the questionnaire score is less than the upper threshold of the preset threshold or greater than the lower threshold, the abnormal detection result representing the target questionnaire data as non-abnormal questionnaire data is determined.
[0095] For example, the first total number of groups represents the target number of groups, and the second total number of groups represents the number of groups with total satisfaction. Since the target number of groups may be inconsistent with the number of groups with total satisfaction, the first ratio and the second ratio can be calculated first, that is, the data can be normalized for comparison. Then, the first ratio is divided by the second ratio to obtain the questionnaire score corresponding to the target questionnaire data, and finally the questionnaire score is compared with the preset threshold to determine the abnormality detection result of the target questionnaire data.
[0096] It should be understood that when the questionnaire score is closer to 1 (that is, the ratio of the first ratio to the second ratio is closer to 1:1), the sub-satisfaction and total satisfaction are more reasonable. Taking the satisfaction score of 1-10 as an example, the sub-satisfaction of each sub-survey item is basically 6 or 7 points, and the total satisfaction is 6 points. On the contrary, if the questionnaire score is farther away from 1, the sub-satisfaction and total satisfaction are more unreasonable. The sub-satisfaction of each sub-survey item is basically 1 point, and the total satisfaction is 10 points. Based on this principle, preset thresholds can be set according to needs. For example, if you want to delete questionnaire data with sub-satisfaction of 1 and total satisfaction of 10 or total satisfaction of 1 and sub-satisfaction of 10, you can set the upper threshold to 10 and the lower threshold to 0.1. If you want to delete questionnaire data with sub-satisfaction of 2 and total satisfaction of 10, or sub-satisfaction of 10 and total satisfaction of 2, you can further narrow the preset threshold range, and so on. The present disclosure does not limit this.
[0097] By adopting the above method, the target classification model is used to classify and predict the target sub-satisfaction to obtain the target classification result, and then the ratio of the first group where the target classification result is located to the second group where the target total satisfaction is located is compared with the preset score threshold, and the data with the farthest distance between the first group and the second group is determined as abnormal questionnaire data, thereby realizing anomaly detection for ordinal type data such as satisfaction. In addition, in the subsequent data statistical analysis process, abnormal data can be eliminated according to the anomaly detection results, thereby improving the reliability of data analysis.
[0098] In possible ways, refer to Figure 2, the disclosed embodiment provides a complete processing flow from questionnaire design to questionnaire data anomaly detection. First, the questionnaire is designed according to the application scenario and questionnaire requirements, and then the questionnaire can be distributed through text messages, the Internet, etc., and the completed questionnaires can be collected. Secondly, the questionnaire data can be preliminarily cleaned to obtain the target questionnaire data, such as setting strong cleaning rules. A classification model is built based on the target questionnaire data, and the sub-satisfaction of the target questionnaire data is input into the built classification model for model prediction to obtain the target classification result. The model AUC index is calculated based on the target classification result, and the first grouping of the target classification result is determined according to the AUC index. The target score of the target questionnaire data is determined based on the ratio of the first grouping to the second grouping of the total satisfaction, and the abnormal data is defined based on the target score and the preset score threshold, and the anomaly detection result of the target questionnaire data is determined.
[0099] By adopting the above method, a method for detecting abnormal data in satisfaction survey questionnaires is provided, which can determine the abnormal questionnaires in the satisfaction survey questionnaires and delete the abnormal questionnaires, which helps to avoid biased conclusions in the subsequent statistical analysis, improve the reliability of data analysis, and improve the robustness of the subsequent data analysis model construction.
[0100] Figure 3 is a block diagram of a data detection device according to an exemplary embodiment. Figure 3 The device 300 includes an acquisition module 301, an input module 302, a grouping module 303 and a determination module 304.
[0101] The acquisition module 301 is configured to acquire target questionnaire data, wherein the target questionnaire data includes target sub-satisfaction and target total satisfaction.
[0102] The input module 302 is configured to input the target sub-satisfaction of the target questionnaire data into the target classification model to obtain the target classification result corresponding to the target questionnaire data, and the target classification result is used to characterize the classification probability corresponding to the predicted total satisfaction of the target questionnaire data.
[0103] The grouping module 303 is configured to determine a first group corresponding to the target classification result and a second group corresponding to the target total satisfaction according to a preset grouping rule.
[0104] The determination module 304 is configured to determine an abnormality detection result of the target questionnaire data according to the first grouping and the second grouping.
[0105] By using the above device, the target classification model is used to classify and predict the target sub-satisfaction to obtain the target classification result, and then the abnormal detection result of the target questionnaire data is determined according to the group where the target classification result is located and the group where the target total satisfaction is located, thereby realizing abnormal detection for the ordered type data such as satisfaction. In addition, in the subsequent data statistical analysis process, abnormal data can be eliminated according to the abnormal detection results, thereby improving the reliability of data analysis.
[0106] Optionally, there are multiple target questionnaire data, and the grouping module 303 includes:
[0107] A first determination submodule is configured to determine the number of target groups according to multiple target classification results corresponding to multiple target questionnaire data;
[0108] A sorting module is configured to sort the multiple target classification results according to a preset order to obtain a target sequence classification result;
[0109] The division module is configured to divide the target sequential classification results based on the number of target groups and determine a first group corresponding to each of the target classification results.
[0110] Optionally, the first determining submodule is configured to:
[0111] Determine the target index value of the target classification model according to the target classification result;
[0112] Multiply the maximum value of the value range of the target total satisfaction by the reciprocal of the target indicator value to obtain the initial number of groups;
[0113] The initial number of groups is rounded up to obtain the target number of groups.
[0114] Optionally, the grouping module 303 is configured to:
[0115] According to the target total satisfaction and the preset correspondence between the total satisfaction and the grouping, a second grouping corresponding to the target total satisfaction is determined.
[0116] Optionally, the first group and the second group have corresponding group sequence numbers, and the determining module 304 includes:
[0117] A second determination submodule is configured to divide the group number corresponding to the first group by the group number corresponding to the second group to obtain a target score corresponding to the target questionnaire data;
[0118] The third determination submodule is configured to determine the abnormality detection result of the target questionnaire data according to the size relationship between the target score and the preset score threshold.
[0119] Optionally, the preset score threshold is determined in the following manner:
[0120] Determine the target index value of the target classification model according to the target classification result;
[0121] Multiply the maximum value of the target total satisfaction value range by the inverse of the target indicator value to obtain the upper limit threshold of the preset score threshold, and use the inverse of the maximum value of the target total satisfaction value range as the lower limit threshold of the preset score threshold.
[0122] Optionally, the third determining submodule is configured to:
[0123] When the target score is equal to the upper threshold or the lower threshold of the preset score threshold, an abnormality detection result is determined that characterizes the target questionnaire data as abnormal questionnaire data.
[0124] Optionally, the target classification model is constructed in the following manner:
[0125] Determining an initial classification model and sample questionnaire data, wherein the sample questionnaire data includes sample sub-satisfaction and sample total satisfaction;
[0126] Determine a sample label of the sample questionnaire data according to a preset label threshold and a sample total satisfaction level of the sample questionnaire data;
[0127] The initial classification model is trained based on the sample labels and the sample sub-satisfaction of the sample questionnaire data to obtain the target classification model after model training.
[0128] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0129] Based on the same inventive concept, a data detection device includes:
[0130] processor;
[0131] a memory for storing processor-executable instructions;
[0132] Wherein, the processor is configured to:
[0133] Obtaining target questionnaire data, wherein the target questionnaire data includes target sub-satisfaction and target total satisfaction;
[0134] Inputting the target sub-satisfaction of the target questionnaire data into the target classification model to obtain the target classification result corresponding to the target questionnaire data, wherein the target classification result is used to characterize the classification probability corresponding to the predicted total satisfaction of the target questionnaire data;
[0135] According to a preset grouping rule, determining a first group corresponding to the target classification result and a second group corresponding to the target total satisfaction;
[0136] An abnormality detection result of the target questionnaire data is determined according to the first grouping and the second grouping.
[0137] Based on the same inventive concept, an embodiment of the present disclosure provides a computer-readable storage medium having computer program instructions stored thereon, which implement the steps of any one of the data detection methods when executed by a processor.
[0138] Figure 4 4 is a block diagram of a data detection device 400 according to an exemplary embodiment. For example, the device 400 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.
[0139] Reference Figure 4 , the device 400 may include one or more of the following components: a processing component 402 , a memory 404 , a power component 406 , a multimedia component 408 , an audio component 410 , an input / output interface 412 , a sensor component 414 , and a communication component 416 .
[0140] The processing component 402 generally controls the overall operation of the device 400, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 402 may include one or more processors 420 to execute instructions to complete all or part of the steps of the above-mentioned data detection method. In addition, the processing component 402 may include one or more modules to facilitate the interaction between the processing component 402 and other components. For example, the processing component 402 may include a multimedia module to facilitate the interaction between the multimedia component 408 and the processing component 402.
[0141] The memory 404 is configured to store various types of data to support operations on the device 400. Examples of such data include instructions for any application or method operating on the device 400, contact data, phone book data, messages, pictures, videos, etc. The memory 404 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0142] The power supply component 406 provides power to the various components of the device 400. The power supply component 406 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device 400.
[0143] The multimedia component 408 includes a screen that provides an output interface between the device 400 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 408 includes a front camera and / or a rear camera. When the device 400 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.
[0144] The audio component 410 is configured to output and / or input audio signals. For example, the audio component 410 includes a microphone (MIC), and when the device 400 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 404 or sent via the communication component 416. In some embodiments, the audio component 410 also includes a speaker for outputting audio signals.
[0145] The input / output interface 412 provides an interface between the processing component 402 and the peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.
[0146] The sensor assembly 414 includes one or more sensors for providing various aspects of the status assessment of the device 400. For example, the sensor assembly 414 can detect the open / closed state of the device 400, the relative positioning of components, such as the display and keypad of the device 400, and the sensor assembly 414 can also detect the position change of the device 400 or a component of the device 400, the presence or absence of user contact with the device 400, the orientation or acceleration / deceleration of the device 400, and the temperature change of the device 400. The sensor assembly 414 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 414 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 414 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
[0147] The communication component 416 is configured to facilitate wired or wireless communication between the device 400 and other devices. The device 400 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 416 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 416 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.
[0148] In an exemplary embodiment, the device 400 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above-mentioned data detection method.
[0149] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 404 including instructions, and the instructions can be executed by a processor 420 of the device 400 to perform the above data detection method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0150] In another exemplary embodiment, a computer program product is also provided. The computer program product includes a computer program executable by a programmable device, and the computer program has a code portion for executing the above data detection method when executed by the programmable device.
[0151] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the present disclosure. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.
[0152] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims
1. A data detection method, characterized in that: include: Obtaining target questionnaire data, wherein the target questionnaire data includes target sub-satisfaction and target total satisfaction; Inputting the target sub-satisfaction of the target questionnaire data into the target classification model to obtain the target classification result corresponding to the target questionnaire data, wherein the target classification result is used to characterize the classification probability corresponding to the predicted total satisfaction of the target questionnaire data; According to a preset grouping rule, determining a first group corresponding to the target classification result and a second group corresponding to the target total satisfaction; An abnormality detection result of the target questionnaire data is determined according to the first grouping and the second grouping.
2. The method according to claim 1, characterized in that: The number of the target questionnaire data is multiple, and determining the first group corresponding to the target classification result according to the preset grouping rule includes: Determine the number of target groups according to multiple target classification results corresponding to multiple target questionnaire data; Sorting the multiple target classification results according to a preset order to obtain a target sequence classification result; The target sequential classification results are divided based on the number of target groups, and a first group corresponding to each target classification result is determined.
3. The method according to claim 2, characterized in that The step of determining the number of target groups according to the multiple target classification results corresponding to the multiple target questionnaire data includes: Determine the target index value of the target classification model according to the target classification result; Multiply the maximum value of the value range of the target total satisfaction by the reciprocal of the target indicator value to obtain the initial number of groups; The initial number of groups is rounded up to obtain the target number of groups.
4. The method according to claim 1, characterized in that Determining the second group corresponding to the target total satisfaction level according to a preset grouping rule includes: According to the target total satisfaction and the preset correspondence between the total satisfaction and the grouping, a second grouping corresponding to the target total satisfaction is determined.
5. The method according to any one of claims 1 to 4, characterized in that: The first group and the second group have corresponding group serial numbers, and determining the abnormality detection result of the target questionnaire data according to the first group and the second group includes: Dividing the group number corresponding to the first group by the group number corresponding to the second group to obtain a target score corresponding to the target questionnaire data; The abnormality detection result of the target questionnaire data is determined according to the magnitude relationship between the target score and the preset score threshold.
6. The method according to claim 5, characterized in that The preset score threshold is determined in the following way: Determine the target index value of the target classification model according to the target classification result; Multiply the maximum value of the target total satisfaction value range by the inverse of the target indicator value to obtain the upper limit threshold of the preset score threshold, and use the inverse of the maximum value of the target total satisfaction value range as the lower limit threshold of the preset score threshold.
7. The method according to claim 5, characterized in that The determining, according to the magnitude relationship between the target score and the preset score threshold, the abnormality detection result of the target questionnaire data includes: When the target score is equal to the upper threshold or the lower threshold of the preset score threshold, an abnormality detection result is determined that characterizes the target questionnaire data as abnormal questionnaire data.
8. The method according to any one of claims 1 to 4, characterized in that: The target classification model is constructed in the following way: Determining an initial classification model and sample questionnaire data, wherein the sample questionnaire data includes sample sub-satisfaction and sample total satisfaction; Determine a sample label of the sample questionnaire data according to a preset label threshold and a sample total satisfaction level of the sample questionnaire data; The initial classification model is trained based on the sample labels and the sample sub-satisfaction of the sample questionnaire data to obtain the target classification model after model training.
9. A data detection device, characterized in that: The device comprises: An acquisition module is configured to acquire target questionnaire data, wherein the target questionnaire data includes target sub-satisfaction and target total satisfaction; An input module is configured to input the target sub-satisfaction of the target questionnaire data into a target classification model to obtain a target classification result corresponding to the target questionnaire data, wherein the target classification result is used to characterize a classification probability corresponding to the predicted total satisfaction of the target questionnaire data; A grouping module, configured to determine a first group corresponding to the target classification result and a second group corresponding to the target total satisfaction according to a preset grouping rule; A determination module is configured to determine an abnormality detection result of the target questionnaire data according to the first grouping and the second grouping.
10. A data detection device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to: Obtaining target questionnaire data, wherein the target questionnaire data includes target sub-satisfaction and target total satisfaction; Inputting the target sub-satisfaction of the target questionnaire data into the target classification model to obtain the target classification result corresponding to the target questionnaire data, wherein the target classification result is used to characterize the classification probability corresponding to the predicted total satisfaction of the target questionnaire data; According to a preset grouping rule, determining a first group corresponding to the target classification result and a second group corresponding to the target total satisfaction; An abnormality detection result of the target questionnaire data is determined according to the first grouping and the second grouping.
11. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the steps of the method described in any one of claims 1 to 8 are implemented.