Online investigation sample life cycle health degree evaluation method and system

Abstract

The invention belongs to the field of data processing and quality control of online questionnaires. The invention provides an online investigation sample life cycle health degree evaluation method, which comprises the following steps: establishing a hierarchical structure: the highest layer is the health degree of each sample of an online investigation activity, the middle layer is a plurality of quality control indexes, and the bottommost layer is the samples participating in the online investigation activity; constructing a judgment matrix, and evaluating the relative importance of the plurality of quality control indexes; calculating a weight vector for the judgment matrix; total ranking of the levels and consistency checking of the levels: calculating the weights of all factors of a certain level for the relative importance of the highest level; if the consistency check is not satisfied, adjusting the judgment matrix; and according to the weights of the plurality of quality control indexes and the qualified rate data of each index, calculating a sample health degree index, and performing evaluation. The invention further provides an online investigation sample life cycle health degree evaluation system. According to the invention, the quality and reliability of research data can be improved.

Description

Technical Field

[0001] This invention relates to the field of data processing and quality control of online survey questionnaires, and specifically to a method and system for assessing the life cycle health of online survey samples based on the Analytic Hierarchy Process (AHP). Background Technology

[0002] During the collection of online survey questionnaires, problems such as abnormal response speed, high answer similarity, duplicate IP addresses, and bots manipulating scores often occur, all of which seriously affect the quality and reliability of the survey results. Currently, although some quality control methods exist, most are limited to single indicators and cannot comprehensively assess the quality of the sample. Summary of the Invention

[0003] To address the problems in the background technology, this invention proposes a method for assessing the lifecycle health of online survey samples, comprising: establishing a hierarchical structure: S1, the highest level is the health level of each sample in the online survey activity, the middle level is multiple quality control indicators, and the lowest level is the samples participating in the online survey activity; S2, constructing a judgment matrix to assess the relative importance of the multiple quality control indicators; S3, calculating a weight vector for the judgment matrix; S4, overall hierarchical ranking and consistency test: calculating the weights of the relative importance of all factors at a certain level to the highest level; S5, based on the results of steps S3 and S4, if the consistency test is not satisfied, adjusting the judgment matrix and repeating steps S3 and S4; S6, calculating and assessing the sample health index based on the weights of the multiple quality control indicators and the pass rate data of each indicator.

[0004] This invention also proposes an online survey sample lifecycle health assessment system, including a computer executable program that, when the program runs, implements the method described above.

[0005] This invention implements quality control of online survey questionnaires by constructing multiple evaluation models and establishes a sample life cycle health index by combining an AHP (Analytic Hierarchy Process) comprehensive evaluation model, which can effectively improve the quality and reliability of survey data.

[0006] In one embodiment, the multi-dimensional comprehensive evaluation model of the present invention not only considers traditional single indicators such as answering speed and answer similarity, but also introduces dimensions such as IP address analysis, account behavior analysis, background information logic verification and open-ended question completion quality, forming a comprehensive and detailed online survey questionnaire quality control system.

[0007] In one embodiment, the present invention combines the Analytic Hierarchy Process (AHP) with quality control and sample health assessment of online survey questionnaires. The AHP method can systematically handle the complex relationships between multiple indicators and calculate a comprehensive score based on the relative importance of each indicator, thereby improving the accuracy and scientific rigor of quality control.

[0008] In one embodiment, the multiple prediction models of the present invention (such as the prediction model of multiple accounts signing in / answering questions with the same IP address, the prediction model of answering questions with virtual IP addresses of the same account, and the prediction model of robot signing in and scoring) can predict and identify possible cheating behaviors in advance, thereby intervening in the data collection stage and further improving the quality of the data.

[0009] In one embodiment, the sample lifecycle health index proposed in this invention is an innovation. The sample lifecycle health index not only considers the quality of the sample at a certain moment, but also comprehensively considers the performance of the sample throughout the entire survey process, thereby enabling a more comprehensive and accurate assessment of the sample quality.

[0010] In one embodiment, the present invention establishes a blacklist / whitelist database management mechanism based on comprehensive evaluation results. By updating and dynamically adjusting the blacklist / whitelist database in real time, low-quality samples can be effectively identified and excluded, ensuring the accuracy and validity of the survey data.

[0011] In one embodiment, the quality control system of the present invention has high flexibility and scalability. As the online survey environment continues to change and new cheating methods emerge, new evaluation indicators and predictive models can be easily added within the existing framework to adapt to new needs and challenges. Attached Figure Description

[0012] To facilitate understanding of the invention, it will be described in more detail with reference to the specific embodiments shown in the accompanying drawings. These drawings depict only typical embodiments of the invention and should not be considered as limiting the scope of protection of the invention.

[0013] Figure 1 This is a flowchart illustrating one embodiment of the method of the present invention.

[0014] Figure 2 This is a flowchart illustrating one embodiment of the method of the present invention.

[0015] Figure 3 This is a graph showing the lifecycle health score of a sample in one embodiment. Detailed Implementation

[0016] The embodiments of the present invention will be described below with reference to the accompanying drawings, so that those skilled in the art can better understand the present invention and implement it. However, the listed embodiments are not intended to limit the present invention. Without conflict, the following embodiments and the technical features in the embodiments can be combined with each other, and the same components are denoted by the same reference numerals.

[0017] The online research sample life cycle health assessment system of the present invention includes multiple assessment models, which conduct all-round quality control on online research questionnaires, and establish sample life cycle health indicators and black and white list libraries in combination with the AHP comprehensive evaluation model to improve the quality and reliability of research results.

[0018] The system of the present invention includes a multi-account sign-in and answering prediction model, a same-account answering prediction model, and an AHP comprehensive evaluation model.

[0019] Specifically, the multi-account sign-in prediction model analyzes the associated data of IP addresses and accounts to determine whether there are abnormal behaviors of multiple accounts under the same IP address, so as to identify possible brushing behaviors. Specifically, the multi-account sign-in and answering prediction model performs the following operations.

[0020] 1) Data grouping and sorting: Group the answering data of each week according to the IP address, and default to sort according to the answering time.

[0021] 2) Answering time interval calculation: For different user IDs (accounts) under the same IP address, calculate their answering time intervals. Let the answering time of the i-th account be (T_i), then the answering time interval between the i-th and j-th accounts is (\Delta T_{ij} = |T_i - T_j|). 3) Abnormal detection: Set a threshold (t_{threshold}) (for example, 60 seconds). When (\Delta T_{ij} < t_{threshold}), it is marked as abnormal, and there may be a behavior of multiple accounts brushing answers.

[0022] Specifically, the same-account answering prediction model analyzes the associated data of accounts and IP addresses to predict whether there is a situation where the same account uses a virtual IP address to answer questions, so as to identify possible cheating behaviors. Specifically, the virtual IP address answering prediction model performs the following operations.

[0023] 1) Data sorting: Sort the answering data of different accounts according to the answering start time.

[0024] 2) Calculation of the time interval between IP addresses: For the answering data of the same account under different IP addresses, calculate the time interval of answering. Let the answering time of account A under the \(i\)-th IP address be \(T_i^A\), then the time interval between the \(i\)-th and \(j\)-th IP addresses of account A is \(\Delta T_{ij}^A = |T_i^A - T_j^A|\).

[0025] 3) Anomaly detection: Similarly, set the threshold \(t_{threshold}\). When \(\Delta T_{ij}^A < t_{threshold}\), it is marked as an anomaly, and there may be behaviors such as using a proxy server or virtual machine. In addition, if there are consecutive "diagonal features" in the matrix (i.e., multiple adjacent \(\Delta T_{ij}^A < t_{threshold}\)), the judgment of abnormal behavior is further strengthened.

[0026] Specifically, the AHP comprehensive evaluation model constructs a comprehensive evaluation model according to the output results of the above multiple evaluation models, combines the AHP analytic hierarchy process, calculates the life cycle health index of each sample, and establishes a black and white list library based on this. Based on this, by building the sample life cycle health index, from the first registration, login to long-term participation in online research activities, comprehensively evaluate and master the health degree and change trend of the sample participating in online research activities (the health degree is represented by a score from 0 to 100, and the higher the score, the higher the health degree of the sample participating in filling out the questionnaire), realize the hierarchical management of sample quality, and provide a decision-making basis for the implementation and requirement matching of subsequent research projects.

[0027] The AHP comprehensive evaluation model uses the AHP analytic hierarchy process as the basic model of the sample life cycle health index. The analytic hierarchy process refers to regarding a complex multi-objective decision-making problem as a system, decomposing the goal into multiple goals or criteria, and further decomposing it into several levels of multiple indicators (or criteria, constraints). Through the qualitative index fuzzy quantification method, calculate the single-level sorting (weights) and total sorting of the levels to be used as a systematic method for multi-objective (multi-index) and multi-scheme optimization decisions. The analytic hierarchy process decomposes the decision-making problem into different hierarchical structures in the order of the overall goal, each level of sub-goals, evaluation criteria, and specific alternative investment plans. Then, use the method of solving the eigenvector of the judgment matrix to obtain the priority weight of each element at each level for a certain element at the previous level. Finally, use the method of weighted sum to hierarchically merge the final weights of each alternative plan for the overall goal. The alternative plan with the largest final weight is the optimal plan.

[0028] The first implementation mode

[0029] In this implementation mode, as Figure 1 shown, the AHP comprehensive evaluation model of the present invention includes the following technical means.

[0030] S1. Establish a hierarchical structure (determine the indicator system): Divide the decision-making objectives, factors considered (decision criteria), and decision objects into three levels—the highest, middle, and lowest—based on their interrelationships, and draw a hierarchical structure diagram. The highest level refers to the purpose of the decision and the problem to be solved. The lowest level refers to the alternative solutions during the decision-making process. The middle level refers to the factors considered and the decision criteria. For adjacent levels, the higher level is called the objective level, and the lower level is called the factor level.

[0031] S2, Constructing a Judgment (Pairwise Comparison) Matrix: When determining the weights between factors at each level, qualitative results are often difficult to accept. Therefore, this invention employs a consistent matrix method, which compares each factor pairwise instead of all together. A relative scale is used to minimize the difficulty of comparing factors with different characteristics, thereby improving accuracy. For example, for a given criterion, each option is compared pairwise, and its importance is assessed. ij Table 1 lists the nine importance levels and their assigned values ​​given by the Saaty scaling method, illustrating the importance comparison results between element i and element j. The matrix formed by the pairwise comparison results is called the judgment matrix. The judgment matrix has the following properties:

[0032]

[0033] Determine matrix element a ij The scaling methods are shown in Table 1.

[0034] Table 1 Proportional Scale Table

[0035] Factor i compared to factor j Quantized value Equally important 1 Slightly important 3 Stronger and more important 5 Strongly important 7 Extremely important 9 The median value between two adjacent judgments 2，4，6，8

[0036] S3, Calculate the weight vector (hierarchical single sorting and its consistency test): corresponding to the largest eigenvalue λ of the judgment matrix. max The eigenvectors of A, after normalization (making the sum of all elements in the vector equal to 1), are denoted as W. The elements of W represent the relative importance weights of factors at the same level relative to a factor at the next higher level. This process of determining the weights is called hierarchical single ranking. Whether hierarchical single ranking can be confirmed requires a consistency test, which refers to determining the acceptable range of inconsistencies in A. The unique non-zero eigenvalue of an n-order consistent matrix is ​​n; the largest eigenvalue of an n-order positive reciprocal matrix A is λ. max ≥n, if and only if λ max When A = n, A is a uniform matrix.

[0037] Since λ continuously depends on a ijThe greater the difference between λ and n, the more severe the inconsistency of A. The consistency index is calculated using CI; the smaller the CI, the greater the consistency. The eigenvector corresponding to the largest eigenvalue is used as the weight vector for the degree of influence of the compared factor on a higher-level factor. The greater the degree of inconsistency, the greater the judgment error. Therefore, the magnitude of λ-n can be used to measure the degree of inconsistency of A. The consistency index is defined as:

[0038]

[0039] λ max The largest eigenvalue of the judgment matrix is ​​represented by , and n represents the order of the judgment matrix (i.e., the number of rows or columns of the matrix).

[0040] CI = 0 indicates perfect consistency; CI close to 0 indicates satisfactory consistency; the larger the CI, the more severe the inconsistency.

[0041] To measure the magnitude of CI, the random consistency metric RI is introduced:

[0042]

[0043] CI1, CI2, ..., CI n This represents the consistency index of each random judgment matrix, where n represents the number of random judgment matrices.

[0044] Among them, the random consistency index RI is related to the order of the judgment matrix. Generally, the larger the order of the matrix, the greater the possibility of random deviations in consistency. The corresponding relationship is shown in Table 2.

[0045] Table 2 Standard values ​​of the average random consistency index RI

[0046] Matrix order 1 2 3 4 5 6 7 8 9 10 RI 0 0 0.58 0.90 1.12 1.24 1.32 1.41 1.45 1.49

[0047] Note: Different standards may have slightly different RI values.

[0048] Considering that deviations in consistency may be due to random causes, when verifying whether the judgment matrix has satisfactory consistency, it is also necessary to compare CI with the random consistency index RI to obtain the test coefficient CR, as shown in the following formula:

[0049]

[0050] Generally, if CR < 0.1, the judgment matrix is ​​considered to have passed the consistency test; otherwise, it does not have satisfactory consistency.

[0051] S4, Overall Hierarchical Ranking and Consistency Test: Calculating the weights of the relative importance of all factors at a certain level to the highest level (overall goal) is called the overall hierarchical ranking. This process is carried out sequentially from the highest level to the lowest level.

[0052] S5. Based on the results of steps S3 and S4, if the consistency check is not satisfied, adjust the judgment matrix and continue to repeat steps S3 and S4.

[0053] Second Implementation Method

[0054] In this embodiment, the AHP comprehensive evaluation model of the present invention implements the following steps.

[0055] S1. Establish a hierarchical structure: the top layer represents the health status of each sample in the online survey activity, the middle layer consists of 6 quality control indicators, and the bottom layer represents the samples participating in the online survey activity.

[0056] Specifically, the purpose of the top-level decision-making process is to evaluate the health status of each sample from their first login and participation in the online survey, using a score range of 0-100. The higher the score, the higher the health status, and the lower the score, the lower the health status.

[0057] The factors considered in the middle layer and the decision-making criteria are based on six quality control indicators of "online questionnaire survey quality control" adopted in this invention: whether the IP address is unique, whether it is a "wool party" (a group of people who exploit loopholes for personal gain), whether it is a bot, whether there are logical errors, whether the answering speed is too fast, and whether the basic information is inconsistent.

[0058] The lowest level consists of the samples that actually participated in the online survey activities.

[0059] When selecting quality control indicators, this invention fully considers various potential problems that may be encountered during the online survey questionnaire data collection process and their impact on the survey results.

[0060] (1) Is the IP address unique?

[0061] An IP address is a unique identifier for a user in cyberspace. In online surveys, if multiple accounts frequently answer questions under a single IP address, it's highly likely that this involves bots, proxy answers, or other cheating activities. These behaviors severely distort the authenticity of the survey data and affect the final analysis results. By detecting the uniqueness of IP addresses, we can effectively identify and exclude samples that use the same IP address for cheating, thereby ensuring the purity and reliability of the survey data.

[0062] (2) Are you a "wool party" (someone who takes advantage of freebies)?

[0063] "Wool-gathering" refers to a group of users who specifically exploit platform loopholes, promotional activities, or other means to gain benefits. In online surveys, wool-gatherers may obtain rewards through fake registration, rapid answering of questions, or other means, without caring about the survey content itself. This behavior not only wastes survey resources but may also distort the data. Identifying and excluding wool-gatherer samples can ensure that survey data comes from respondents who genuinely care about the survey content and are willing to provide valuable information, thereby improving the authenticity and validity of the data.

[0064] (3) Is it a robot?

[0065] Robots using automated responses are a common cheating method in online surveys. While robots can complete questionnaires quickly and accurately, they cannot provide genuine market insights or consumer feedback. This automated data has no real value to the survey results and may even mislead the analysis. By detecting robot behavior patterns, such as abnormal response speed and formulaic answers, robot samples can be effectively identified and excluded, thus ensuring the authenticity and reliability of the survey data.

[0066] (4) Are there any logical errors?

[0067] Questionnaire questions are often logically related. If respondents' answers are contradictory or logically flawed, it's likely due to random answering, proxy answering, or careless reading of the questions. Such careless responses reduce the quality and value of the data. By detecting logical errors in questionnaires, we can identify respondents who carefully read the questions, thought critically, and answered them, thereby improving the accuracy and reliability of the data.

[0068] (5) Is the answering speed too fast?

[0069] While fast response time doesn't necessarily indicate low-quality answers, extremely rapid response is often associated with cheating. For example, bots or proxy test-takers might complete a large number of questionnaires in a short period to earn rewards or complete tasks. By setting reasonable thresholds for response speed, samples with abnormally fast response times can be identified and further reviewed or excluded. This helps ensure the authenticity and reliability of survey data.

[0070] (6) Is the basic information inconsistent?

[0071] Basic information (such as age, gender, and occupation) is a crucial component of questionnaire design. If respondents' answers do not match this basic information, it is likely due to false reporting or arbitrary completion. Such inaccurate basic information can negatively impact subsequent data analysis and market insights. By checking the consistency of basic information, we can ensure that respondents' answers are based on their true circumstances, thereby improving the accuracy and reliability of the data.

[0072] In summary, the selection of these quality control indicators is based on an in-depth analysis and understanding of various issues that may be encountered during the online survey questionnaire data collection process. Together, they constitute a comprehensive and meticulous quality control system aimed at ensuring the authenticity and reliability of the survey data. Furthermore, when constructing an online survey sample lifecycle health assessment system based on the Analytic Hierarchy Process (AHP), the correlation between the quality control indicators is a crucial factor that cannot be ignored. These indicators exhibit strong correlations.

[0073] (1) Mutual constraint

[0074] There are certain interrelationships among different indicators. For example, answering questions too quickly may increase the similarity of answers, because fast-responding individuals often lack sufficient time to formulate and edit their answers, thus tending to choose similar or templated responses. Similarly, increased similarity in answer content may also lead to a higher rate of logical errors in the questionnaire, as similar answers often lack logical consistency.

[0075] This interdependence reflects the inherent connections and mutual influences among different indicators. When assessing the health of a sample, these interdependencies need to be fully considered to avoid evaluating one indicator in isolation while ignoring the impact of other related indicators.

[0076] (2) Complementarity

[0077] While different indicators are mutually restrictive, they also have a certain degree of complementarity. For example, although answering too quickly may affect the quality of the answers, answering quickly does not necessarily mean that the quality of the answers is low; similarly, although increased similarity in the content of the answers may lead to an increase in the logical error rate of the questionnaire, similar answers may also reflect the common views or cognitions of the respondents in some cases.

[0078] This complementarity reflects the different focuses and advantages of various indicators in assessing the health of a sample. When constructing an assessment system, it is necessary to make full use of these complementarities, combining multiple indicators for comprehensive evaluation to improve the accuracy and reliability of the assessment results.

[0079] (3) Hierarchy

[0080] The hierarchical structure constructed in this invention divides quality control indicators into three levels: the highest level, the middle level, and the lowest level. The highest level represents the decision-making objective (i.e., assessing the health of the sample), the middle level represents the factors considered (i.e., each quality control indicator), and the lowest level represents the decision-making object (i.e., the sample participating in the online survey).

[0081] This hierarchical structure not only reflects the hierarchical relationships between indicators but also facilitates the construction and application of the Analytic Hierarchy Process (AHP) and comprehensive evaluation models. Through the AHP, complex relationships between multiple indicators can be systematically addressed, and a comprehensive score can be calculated based on the relative importance of each indicator. This comprehensive score can more comprehensively reflect the health status of the sample, providing strong decision support for researchers.

[0082] In summary, the correlations among quality control indicators are reflected in three aspects: mutual constraint, complementarity, and hierarchy. These correlations provide an important theoretical foundation and practical guidance for constructing an online survey sample lifecycle health assessment system based on the Analytic Hierarchy Process (AHP). When constructing the assessment system, these correlations need to be fully considered, and multiple indicators should be combined for comprehensive evaluation to improve the accuracy and reliability of the assessment results.

[0083] S2. Construct a judgment matrix to assess the relative importance of the above 6 quality control indicators. The assessment results are shown in Table 3.

[0084] Table 3. AHP (Analog-HP) Judgment Matrix for Each Quality Control Indicator

[0085]

[0086] S3, using the AHP hierarchical method (calculation method: sum-product method) to construct a 6th-order judgment matrix based on 6 factors including user IP address, risk control of "wool party" (a term referring to exploiting promotional offers), bots, pre- and post-logic logic, answering speed, and basic information, yields the following feature vector:

[0087] The values ​​are (0.709, 2.317, 1.242, 0.320, 0.829, 0.583), and the corresponding weights for the six items are 11.821%, 38.613%, 20.695%, 5.337%, 13.820%, and 9.714%, respectively. In addition, the largest eigenvalue (6.569) can be calculated from the eigenvectors. Then, the CI value (0.114) is calculated using the largest eigenvalue, where CI = [(largest eigenvalue - n) / (n-1)]. The CI value is used for subsequent consistency checks.

[0088] Table 3.3 Results of AHP (Analog-Hierarchical Analysis)

[0089]

[0090] This invention constructs a 6th-order judgment matrix, with a random consistency RI value of 1.260, which is used for subsequent consistency verification calculations.

[0091] Table 4 Summary of Consistency Test Results

[0092] Maximum eigenvalue CI value RI value CR value Consistency test results 6.569 0.114 1.260 0.090 pass

[0093] Generally, a smaller CR value indicates better consistency of the judgment matrix. Typically, a CR value less than 0.1 indicates that the judgment matrix meets the consistency test; if the CR value is greater than 0.1, it indicates a lack of consistency, and the judgment matrix should be adjusted appropriately before re-analysis. In this study, the calculated CI value for the 6th-order judgment matrix is ​​0.114, and the RI value (from the table) is 1.260. Therefore, the calculated CR value is 0.090 < 0.1, meaning that the judgment matrix in this study meets the consistency test, and the calculated weights are consistent.

[0094] In constructing the judgment matrix of quality control indicators and evaluation matrix, this invention fully considers its rationality and innovation.

[0095] (1) Reasonableness

[0096] Consistency Verification and Adjustment: After constructing the judgment matrix, this invention performs a rigorous consistency verification. By calculating the consistency index (CI) and the random consistency index (RI), and comparing their ratio (CR), the consistency of the judgment matrix is ​​effectively verified. If the CR value is greater than 0.1, the judgment matrix is ​​appropriately adjusted and optimized to ensure that it meets the consistency requirements. This consistency verification and adjustment process improves the rationality and reliability of the judgment matrix.

[0097] (2) Innovation

[0098] Multi-dimensional Comprehensive Evaluation Model: This invention not only considers traditional single indicators such as answering speed and answer similarity, but also introduces dimensions such as IP address analysis, account behavior analysis, background information logical verification, and open-ended question completion quality, constructing a multi-dimensional comprehensive evaluation model. This model can more comprehensively and accurately assess the quality of samples, providing researchers with richer and more valuable information.

[0099] Dynamic Adjustment Mechanism: This invention introduces a dynamic adjustment mechanism during the construction of the judgment matrix. As the online survey environment constantly changes and new cheating methods emerge, new evaluation indicators and prediction models can be easily added to the existing framework to adapt to new needs and challenges. This dynamic adjustment mechanism improves the system's flexibility and scalability, enabling it to better cope with complex and ever-changing online survey environments.

[0100] Innovative technical methods: When constructing the judgment matrix, this invention employs the consistent matrix method (i.e., pairwise comparison) to determine the relative importance of each indicator. This method not only reduces the difficulty of comparing factors with different properties but also improves the accuracy and scientific rigor of the judgment. Furthermore, this invention utilizes the sum-product method to calculate the eigenvectors and weight vectors of the judgment matrix and performs normalization processing, providing strong support for subsequent sample health assessment. These innovative technical methods improve the efficiency and accuracy of judgment matrix construction, laying a solid foundation for the application of the assessment system.

[0101] In summary, this invention fully considers both rationality and innovation when constructing the judgment matrix for quality control indicators and evaluation matrices. By drawing on the results of expert consultation and questionnaire surveys, conducting consistency checks and adjustments, and introducing a multi-dimensional comprehensive evaluation model, a dynamic adjustment mechanism, and innovative technical means, the rationality and innovation of the judgment matrix are ensured. These measures not only improve the accuracy and reliability of the evaluation system but also enhance its flexibility and scalability, enabling it to better cope with complex and ever-changing online survey environments.

[0102] S4. Based on the weights of each quality control indicator and the pass rate data of each indicator, a sample health index is constructed. Table 5 shows the pass rate data of the six quality control indicators of the sample. The statistical method is the number of times the sample is judged as a qualified sample from the first login and the start of the survey activity to the present / the total number of times the sample has participated in the survey.

[0103] Table 5. Pass rates of various indicators of life cycle health in the sample

[0104]

[0105] Therefore, the health score of the sample lifecycle = 100 * (“Questionnaire Logic Weight” * “Questionnaire Logic” pass rate + “Answering Speed” Weight * “Answering Speed” pass rate + “Answering Time Similarity” Weight * “Answering Time Similarity” pass rate + “Answering Content Similarity” Weight * “Answering Content Similarity” pass rate + “Virtual IP Answering” Weight * “Virtual IP Answering” pass rate + “Multiple Accounts / Multiple Devices” Weight * “Multiple Accounts / Multiple Devices” pass rate) Formula (1)

[0106] Corresponding to the life cycle health of Sample 1 in Table 3.1

[0107] =100*(11.82%*95%+38.61%*95%+20.69%*98%+5.34%*95%+13.82%*89%+9.71%*70%)=92.36 (points).

[0108] For example, the sample's health assessment status from initial login and participation in the online survey to the present is as follows: Figure 3 The participation status of each sample in the online survey was comprehensively evaluated, and the samples were evaluated according to the evaluation rules in the table below.

[0109] Table 6. Results of Life Cycle Health Assessment for Samples

[0110]

[0111] The sample lifecycle health index visually displays the quality level of the online survey sample library, providing clear data support for researchers to grasp the overall sample quality status. Furthermore, in subsequent online survey project applications, samples can be matched and screened specifically for online survey projects based on the sample health level to meet project quality requirements.

[0112] This invention applies AHP (Analysis of Hierarchy Process) to the field of questionnaires, but faces a series of challenges and difficulties.

[0113] (1) Determining the weight of indicators

[0114] Challenge: The core of the AHP method lies in determining the relative importance (i.e., weights) of each indicator. However, in the field of questionnaires, the degree of influence of different indicators on survey results is often difficult to quantify directly. For example, the degree of influence of indicators such as the uniqueness of IP addresses, answering speed, and answer similarity on survey results may vary depending on factors such as the purpose of the survey and sample characteristics.

[0115] Determining the weights of each indicator typically requires obtaining subjective judgments through expert consultation, questionnaires, and other methods. However, these subjective judgments often contain elements of subjectivity and uncertainty. Furthermore, due to differences in background and understanding among different experts or respondents, they may have varying opinions on the importance of the same indicator. This increases the difficulty and complexity of determining the weights.

[0116] (2) Consistency test of the judgment matrix

[0117] Challenge: When constructing the judgment matrix, it is necessary to ensure its consistency, meaning the relative importance relationships between the indicators should satisfy transitivity. However, in practice, due to the differences and complexity of subjective judgments, it is difficult to guarantee the complete consistency of the judgment matrix.

[0118] Consistency checks on the judgment matrix are a crucial step in the AHP (Advanced Personality Hierarchy Process) method. If the judgment matrix does not meet consistency requirements, the weights calculated from it will be inaccurate, affecting the final evaluation results. To ensure the consistency of the judgment matrix, consistency checks and adjustments are typically required. However, such adjustments often involve complex mathematical calculations and subjective judgments, increasing the difficulty and complexity of the operation.

[0119] (3) The complexity of data processing

[0120] Challenges: The AHP method involves a significant amount of data processing and computation. In the questionnaire field, due to the large sample size and numerous indicators, data processing is particularly complex. For example, when constructing the judgment matrix, it is necessary to perform pairwise comparisons for each indicator and calculate the corresponding weights and consistency indicators; when calculating the health of the sample, it is necessary to comprehensively consider the performance of multiple indicators and perform operations such as weighted summation.

[0121] The complexity of data processing not only increases the difficulty and time cost of operations but may also affect the accuracy and reliability of evaluation results. To improve data processing efficiency and accuracy, computer technology and algorithm optimization are needed. However, the application of these technologies also requires certain professional knowledge and technical support. To address these challenges, this invention takes the following measures to ensure the effective application of the AHP method in the questionnaire field.

[0122] Expert consultation and questionnaire surveys were introduced: By inviting industry experts, scholars, and practitioners to participate in consultations and surveys, subjective judgment data on the relative importance of each indicator were obtained. This data provided strong support for the construction of the judgment matrix.

[0123] Consistency checks and adjustments are performed: After constructing the judgment matrix, a rigorous consistency check is conducted. If the judgment matrix does not meet the consistency requirements, appropriate adjustments and optimizations are made to ensure its rationality.

[0124] Optimizing with computer technology and algorithms: This involves using computer technology and algorithm optimization techniques to improve the efficiency and accuracy of data processing. For example, programming languages ​​and algorithm libraries can be used to implement operations such as constructing judgment matrices, calculating weights, and performing consistency checks.

[0125] The following describes a specific example.

[0126] 1) Data Collection: Before implementing this invention, it is necessary to collect answer data. This data usually comes from online answering systems and includes, but is not limited to, the following information.

[0127] User ID: A unique identifier for each user, used to track user behavior.

[0128] IP address: The network address used by the user when answering questions, used to identify the user's geographical location and possible proxy usage.

[0129] Answer time: The specific time when a user submits their answer, used to analyze the frequency and time interval between answers.

[0130] Answer content: The answers submitted by users can be used for subsequent quality assessment and content analysis.

[0131] This data can be collected through the backend database of the online quiz system, ensuring the integrity and accuracy of the data.

[0132] 2) Data processing and analysis: After collecting the answer data, the above models will be run for processing and analysis.

[0133] a) Perform IP address-based response time interval analysis, including:

[0134] Data grouping and sorting: Group the answer data according to IP address and sort it according to the answer time.

[0135] Calculate the time interval for answering questions for different user IDs under the same IP address.

[0136] Anomaly detection: Set a threshold, and mark an anomaly when the time interval is less than the threshold.

[0137] b) Perform IP address translation analysis based on user behavior, including:

[0138] Data sorting: Sort the answer data by answer time.

[0139] Calculate the time interval between responses from different IP addresses: For the same user ID, calculate the time interval between responses from different IP addresses.

[0140] Anomaly detection: Similarly, a threshold is set, and when the time interval is less than the threshold or a "diagonal feature" appears, it is marked as an anomaly.

[0141] 3) Conduct sample health assessments and manage blacklists / whitelists / greylists, including:

[0142] a) Sample Health Assessment: During data analysis, the health of the samples can also be assessed. This can be achieved by comprehensively considering multiple indicators, such as response time, similarity of response content, and IP address stability. By setting reasonable evaluation criteria and weights, a health score is calculated for each sample, thereby assessing its quality.

[0143] b) Blacklist / Whitelist / Graylist Management: To more effectively manage abnormal behavior, a blacklist / whitelist / graylist database can be established. Blacklist: Contains user IDs and IP addresses known to exhibit abnormal behavior; these users or IPs will be directly denied access to the quiz system. Whitelist: Contains user IDs and IP addresses known to have normal behavior and good reputation; these users or IPs enjoy higher trust levels and fewer restrictions. Graylist: Contains user IDs and IP addresses whose behavior is suspicious or requires further observation. These users or IPs will have some functions restricted or be subject to stricter monitoring. The blacklist / whitelist / graylist database can be dynamically updated and adjusted according to actual needs to ensure the security and data quality of the quiz system.

[0144] In a preferred embodiment, the system of the present invention further includes: a question-answering speed evaluation model, a question-answering duration similarity model, a question-answering answer similarity model, and a background information logic and main questionnaire logic verification model.

[0145] The answering speed assessment model analyzes respondents' answering time, combines the total questionnaire duration and the number of questions, calculates the average answering speed of respondents, and sets a reasonable speed threshold to identify respondents with abnormal answering speed.

[0146] The response time similarity model compares the response times of respondents and calculates the similarity to identify respondents who may have referenced or plagiarized each other.

[0147] The answer similarity model uses text similarity algorithms to calculate the similarity between respondents' answers in order to identify respondents who may have copied or plagiarized answers.

[0148] The background information logic and main questionnaire logic verification model checks whether the respondents' background information is reasonable and consistent, and whether there are any logical errors or unreasonable situations in the completion of the main questionnaire.

[0149] The specific scheme of the above model has been published in the applicant's invention patent "An Online Survey Sample Quality Assessment System" (Patent No. ZL 2022 1 0360899.2), which is quoted in full in this application for reference.

[0150] This invention employs multiple evaluation models to conduct comprehensive quality control on online survey questionnaires, significantly improving the quality and reliability of survey results. Furthermore, by combining the sample lifecycle health index and blacklist / whitelist database established using the AHP comprehensive evaluation model, low-quality samples can be effectively identified and excluded, enhancing the accuracy and validity of survey data.

[0151] The embodiments described above are merely preferred embodiments of the present invention. The terms "in one embodiment," "in another embodiment," "in yet another embodiment," or "in still another embodiment" used in this specification all refer to one or more of the same or different embodiments according to this disclosure. Ordinary variations and substitutions made by those skilled in the art within the scope of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for assessing the lifecycle health of online survey samples, characterized in that, include: S1. Establish a hierarchical structure: the top layer is the health status of each sample in the online survey activity, the middle layer is multiple quality control indicators, and the bottom layer is the samples that participated in the online survey activity. S2, Construct a judgment matrix to assess the relative importance of the multiple quality control indicators; S3, calculate the weight vector for the judgment matrix; S4, Overall Hierarchical Ranking and Consistency Test: Calculate the weights of the relative importance of all factors at a certain level to the highest level; S5. Based on the results of steps S3 and S4, if the consistency check is not satisfied, adjust the judgment matrix and continue to repeat steps S3 and S4. S6. Calculate and evaluate the sample health index based on the weights of the multiple quality control indicators and the pass rate data of each indicator.

2. The method according to claim 1, characterized in that, Step S4 includes: 1) Hierarchical single ranking and its consistency test: The eigenvector corresponding to the largest eigenvalue of the judgment matrix is ​​normalized. The elements of the eigenvector are the ranking weights of the relative importance of factors at the same level to a factor at the next higher level. 2) Overall hierarchical ranking and consistency test: Calculate the weights of the relative importance of all factors at a certain level to the highest level; 3) If the consistency test is not satisfied, adjust the judgment matrix and continue with the hierarchical single sort and its consistency test, and the hierarchical total sort and its consistency test.

3. The method according to claim 1, characterized in that, The multiple quality control indicators include: Check if the P address is unique, if it's a "wool party" (a group that exploits promotional offers), if it's a bot, if there are logical errors, if the answering speed is too fast, and if the basic information is inconsistent.

4. The method according to claim 2, characterized in that, In step S2, When constructing the judgment matrix, construct a 6th order judgment matrix; The consistency of a matrix is ​​determined by calculating the consistency index CI and the random consistency index RI, and comparing their ratio CR.

5. The method according to claim 2, characterized in that, The eigenvectors are obtained by analyzing the judgment matrix using the AHP hierarchical method.

6. The method according to claim 4, characterized in that, Methods for constructing the judgment matrix include: using the consistent matrix method, comparing each pair of data, and the matrix formed by the pairwise comparison results is called the judgment matrix.

7. The method according to claim 6, characterized in that, The consistency index CI is: λ max The largest eigenvalue of the judgment matrix is ​​represented by , and n represents the order of the judgment matrix (i.e., the number of rows or columns of the matrix). The random consistency index RI is: CI1, CI2, ..., CI n This represents the consistency index of each random judgment matrix, where n represents the number of random judgment matrices.

8. An online survey sample lifecycle health assessment system, characterized in that, This includes computer executable programs that, when run, implement the method described in any one of 1-7 above.

Images