Panel management system and method for online survey sample extraction, quality evaluation and result prediction using the same

Abstract

The panel management system can collect attribute data, including gender, age, residence, education level, income level, occupation, and political orientation, through panel candidate registration information and profiling surveys, and store this data in a panel database. While panel candidates participate in the survey, the system can collect response history data, including response content, response time, and selection patterns, as well as behavioral data. The system can calculate a reliability score for each panel based on the attribute data, response history data, and behavioral data. If the value of attribute data changes within a specified period, the system can detect attribute inconsistencies by determining, based on predefined rules, whether the change is a combination that is realistically impossible to occur. The system can automatically select target panels as a sample by receiving target demographics, required sample size, and minimum reliability criteria as survey request conditions and utilizing the reliability scores for each panel. While collecting survey responses from the sample, the system can calculate a response quality score by evaluating response time, selection patterns, logical consistency, and text quality. The system can control the updating of the reliability score for each panel based on the response quality score.

Description

Technology Field

[0001] The present invention relates to a management technology for online panels used in the fields of online surveys, public opinion polls, and market research.

[0002] More specifically, the invention relates to an artificial intelligence-based panel management system and a method of operation thereof, which integrates and manages information on the demographic attributes of a panel, survey response history, and online behavioral data to quantitatively evaluate panel reliability, automatically extract samples that meet survey request conditions, evaluate the quality of the survey response process in real time, and predict results for similar future surveys in advance based on accumulated data.

[0003] Here, a panel may refer to a group of people who have agreed to participate repeatedly and regularly in specific surveys, research, or service improvement activities. Background Technology

[0005] Recently, non-probability sampling methods using online panels are being widely utilized in various fields, such as market research, public opinion polls, and academic research, due to their advantages of lower survey costs and faster data collection speeds compared to traditional probability sampling methods.

[0006] However, it was common for some respondents to enter false information or answer insincerely while participating in surveys solely for monetary rewards or point accumulation. While existing survey platforms provide some simple filters, such as those for excessively short response times or linear response patterns involving the consecutive selection of the same option, methods to systematically identify and manage dishonest respondents by comprehensively analyzing panel members' long-term response history and online behavioral data are relatively lacking.

[0007] Furthermore, demographic attribute information—such as gender, age, residence, education level, income level, occupation, and political orientation—entered by panelists at the time of registration may become inconsistent with their actual status over time, and the possibility that it was falsely entered from the outset cannot be ruled out. If samples are extracted and analyses are performed under the premise that this attribute information is factual, distortion may occur in the survey results. In particular, there is a problem in effectively filtering out fraudulent respondents due to the lack of a function to automatically detect cases where the same panelist provides different responses for attribute values ​​that are realistically difficult to change rapidly within a short period, such as education level or income level.

[0008] In addition, there are limitations to target extraction methods based on screening questions. For example, when conducting market research on large dog products, conventional systems typically used a method of inserting a screening question (e.g., "Do you currently own a large dog?") at the beginning of the survey and conducting the main survey only on those who responded positively to that question. However, this method has the problem that it is difficult to detect in advance those who falsely respond positively to receive a reward without actually owning a dog, and the time and cost required for the survey increase because the screening question must be presented to a much larger number of respondents to secure a valid sample.

[0009] Furthermore, in many survey services, panel management, survey design, sampling, quality inspection, and data analysis are performed separately within different systems or organizations. As a result, it is difficult to organically link information regarding the panel's long-term response tendencies and reliability to the actual sampling and analysis stages, leading to inefficiencies where information accumulated at each stage cannot be effectively utilized for decision-making in subsequent stages. The problem to be solved

[0011] The present invention aims to provide a panel management system and method that systematically identifies and manages dishonest and fraudulent respondents by integrally collecting and managing demographic attribute data, survey response history data, and online behavioral data of a panel, quantitatively evaluating the reliability of each panel, and classifying it into multiple grades.

[0012] In addition, the present invention aims to effectively filter out false responses and attribute manipulation by performing periodic re-verification on core attributes including gender, age, place of residence, educational background, income level, occupation group, and political orientation, and by automatically detecting cases where the same panel responds with changes in attribute values ​​that cannot realistically occur within a predetermined period, thereby lowering the reliability of the panel or excluding it from sampling.

[0013] The present invention aims to secure a highly reliable target sample with a small number of invitations without separate large-scale screening by analyzing past survey response history and community activity history stored in a panel database, without relying on screening questions, and extracting panels that meet specific target conditions as a pre-validated candidate group. means of solving the problem

[0015] The panel management system can collect attribute data, including gender, age, residence, education level, income level, occupation, and political orientation, through panel candidate registration information and profiling surveys, and store this data in a panel database. While panel candidates participate in the survey, the system can collect response history data, including response content, response time, and selection patterns, as well as behavioral data. The system can calculate a reliability score for each panel based on the attribute data, response history data, and behavioral data. If the value of attribute data changes within a specified period, the system can detect attribute inconsistencies by determining, based on predefined rules, whether the change is a combination that is realistically impossible to occur. The system can automatically select target panels as a sample by receiving target demographics, required sample size, and minimum reliability criteria as survey request conditions and utilizing the reliability scores for each panel. While collecting survey responses from the sample, the system can calculate a response quality score by evaluating response time, selection patterns, logical consistency, and text quality. The system can control the updating of the reliability score for each panel based on the response quality score.

[0016] An online survey sampling, quality evaluation, and result prediction method using a panel management system may include: an action of collecting attribute data including gender, age, residence, education level, income level, occupation group, and political orientation through panel candidate registration information and profiling surveys and storing it in a panel database; an action of collecting response history data and behavioral data including response content, response time, and selection patterns while panel candidates participate in the survey; an action of calculating a reliability score for each panel based on attribute data, response history data, and behavioral data; an action of detecting attribute inconsistencies by determining whether a change is a combination that is not realistically possible according to predefined rules when the value of attribute data changes within a predetermined period; an action of automatically selecting a survey target panel as a sample using the reliability score for each panel by receiving target demographics, required sample size, and minimum reliability criteria as survey request conditions; an action of calculating a response quality score by evaluating response time, selection patterns, logical consistency, and text quality while collecting survey responses from the sample; and an action of updating the reliability score for each panel based on the response quality score. Effects of the invention

[0018] The panel management system according to the present invention and the method for online survey sampling, quality evaluation, and result prediction using the same can improve the overall quality of the panel pool by integrating the panel's attribute data, response history data, and behavioral data to quantitatively evaluate reliability, and by continuously identifying and removing false respondents and attribute manipulators through periodic re-verification of key attributes and detection of temporal discrepancies in attribute values.

[0019] In addition, the present invention utilizes past survey response history and community activity history to extract panels meeting specific target conditions as a pre-verified candidate group, thereby enabling the securing of a highly reliable target sample with a small number of invitations without the need to distribute screening questions, which has the effect of reducing the time and cost required for the survey.

[0020] The present invention has the effect of increasing the reliability of data used for analysis by immediately detecting and removing insincere or fraudulent responses through a comprehensive analysis of response time per item, linear response patterns, logical consistency, and text quality to evaluate response quality in real time and reflecting this in panel reliability.

[0021] The present invention generates a digital twin profile that quantifies the response tendencies of each panel and simulates the results of similar surveys in advance based on this profile, thereby distinguishing between surveys that a survey client should actually conduct and those that should not, and optimizing the sample size, target segments, and scenario priorities when necessary. Brief explanation of the drawing

[0022] FIG. 1 is a drawing for explaining a system utilizing an artificial intelligence model according to one embodiment. FIG. 2 is a diagram illustrating the learning of a neural network according to one embodiment. FIG. 3 is a diagram illustrating the configuration of an artificial intelligence model according to one embodiment. Figure 4 is a block diagram showing the configuration of a panel management system according to one embodiment. FIG. 5 illustrates the overall operation flow of a panel management system according to one embodiment. FIG. 6 illustrates the operation flow of attribute mismatch detection in a panel management system according to one embodiment. FIG. 7 is a flowchart illustrating the operation of calculating a response quality score of a panel management system according to one embodiment. FIG. 8 is a block diagram showing the database structure of a panel management system according to one embodiment. FIG. 9 is a block diagram showing the hierarchical structure of a panel management system according to one embodiment. Specific details for implementing the invention

[0023] Various embodiments are described in detail below with reference to the attached drawings. However, these embodiments may be modified in various ways, and therefore the scope of the patent application is not limited or restricted by these embodiments. All modifications and equivalent substitutes should be considered to be included within the scope of the rights. The specific structural or functional descriptions for each embodiment are merely for illustrative purposes and may be modified in various forms. Accordingly, the embodiments are not limited to a specific form, and the scope of this specification includes variations of the technical concept, equivalents, or substitutes.

[0024] The terms "first" or "second" may be used to describe various components, but are intended merely to distinguish each component. For example, the first component may be named the second component, and conversely, the second component may be named the first component. When a component is referred to as being "connected" to another component, this means that it may be directly connected or that there may be another component in between. The terms used are for descriptive purposes only and are not intended to be limiting. Singular expressions include the plural unless otherwise interpreted in the context. In this specification, expressions such as "comprising" or "having" indicate the presence of the features or elements described in the specification and should be understood as not excluding the presence of other features or elements.

[0025] Unless otherwise specifically defined, all terms used herein have the meanings generally understood by those skilled in the art. Terms defined in commonly used dictionaries should be interpreted in the context of the relevant technology and should not be interpreted in an overly formal sense unless explicitly defined in this application. Additionally, when describing the drawings, the same reference numerals are used for identical components, and redundant descriptions are omitted. When describing embodiments, if a detailed description of the relevant technology would become unnecessarily obscured, such detailed description may be omitted.

[0026] The embodiments can be implemented in various forms, such as personal computers, laptops, tablets, smartphones, televisions, smart home appliances, intelligent vehicles, kiosks, and wearable devices. An artificial intelligence (AI) system is a computer system that implements human-level intelligence; unlike existing rule-based smart systems, it is a system in which the machine learns and makes decisions on its own. As the use of AI systems increases, recognition rates improve and user preferences are understood more accurately, leading to the gradual replacement of existing systems with deep learning-based AI.

[0027] Artificial intelligence technology consists of machine learning and various component technologies that utilize it. Machine learning refers to algorithms that autonomously classify and learn the characteristics of input data, while component technologies are techniques that mimic human cognitive and judgment functions through machine learning algorithms, such as deep learning. These technologies include linguistic understanding, visual understanding, reasoning and prediction, knowledge representation, and motion control.

[0028] The various fields where artificial intelligence technology is applied are as follows. Linguistic understanding refers to the technology of recognizing and processing human language, including natural language processing, machine translation, dialogue systems, question answering, and speech recognition and synthesis. Visual understanding refers to the technology of recognizing and processing objects, including object recognition, tracking, image search, human recognition, and scene understanding. Inference and prediction refer to the technology of judging and predicting information, including knowledge-based inference, optimization prediction, and recommendation systems. Knowledge representation refers to the technology of automatically processing information from human experience, including knowledge construction and management. Motion control refers to the technology of controlling the movement of autonomous vehicles or robots, including navigation, collision avoidance, and driving control. Generally, machine learning algorithms are trained using a trial-and-error method to apply them to real-life situations. Deep learning, in particular, requires hundreds of thousands of iterations. When it is difficult to execute this in a real-world environment, training is conducted through simulations by implementing a virtual environment on a computer.

[0029] In this invention, Artificial Intelligence (AI) refers to a technology that implements human learning, reasoning, and perceptual abilities on a computer, and includes concepts such as machine learning and symbolic logic. Machine learning (ML) is an algorithmic technology that learns the characteristics of input data on its own. Through such machine learning algorithms, AI technology can analyze input data, learn the results, and perform judgments or predictions based on them. Furthermore, technologies that mimic human cognitive and judgment functions using machine learning algorithms can also be included in the category of AI. For example, fields such as linguistic understanding, visual understanding, reasoning and prediction, knowledge representation, and motion control fall into this category. Machine learning refers to the process of training a neural network model through experience in processing data. Through this, computer software can improve its data processing capabilities on its own. A neural network model models the correlations between data and can be expressed by various parameters. The core of machine learning is to optimize the model's parameters by extracting and analyzing features from given data to derive relationships between the data and repeating this process. For example, a neural network model can learn the relationship between input and output, or learn relationships by deriving regularities solely from the input data.

[0030] Artificial intelligence learning models or neural network models are designed to implement the structure of the human brain on a computer and include multiple network nodes that mimic neurons. These nodes are interconnected to exchange signals and process data through layers of varying depths within the AI ​​learning model. Such models may include artificial neural networks and convolutional neural networks (CNNs). For example, AI learning models can be machine learned through methods such as supervised learning, unsupervised learning, and reinforcement learning. Machine learning algorithms that may be used include decision trees, Bayesian networks, support vector machines, artificial neural networks, AdaBoost, perceptrons, genetic programming, and clustering.

[0031] CNNs are a type of multi-layer perceptron designed with minimal preprocessing, consisting of one or more convolutional layers and general artificial neural network layers. Thanks to this structure, CNNs can effectively utilize 2D input data and demonstrate excellent performance in image and speech fields. CNNs are trained via standard backpropagation and have the advantage of being easier to train and using fewer parameters than other feedforward artificial neural network techniques. Convolutional networks are neural networks containing a set of nodes with bound parameters, and many computer vision tasks have been improved as the amount of training data increases and computational power improves. In today's large-scale datasets, overfitting is not critical, and increasing the network size improves test accuracy. Since the optimal use of computing resources becomes a limiting factor, distributed and scalable implementations of deep neural networks become necessary.

[0033] FIG. 1 is a diagram illustrating a system utilizing an artificial intelligence model according to one embodiment.

[0034] As illustrated in FIG. 1, a system (100) utilizing an artificial intelligence model may include multiple user terminals (110-1, ���), a server (120), and a database (130). According to one embodiment, the database (130) is configured separately from the server (120), but it is not necessary to configure it this way, and the database (130) may be located inside the server (120). For example, the server (120) may include multiple artificial intelligences for performing machine learning algorithms. According to another embodiment, multiple user terminals (110-1, ���), a server (120), and a database (130) may be connected to communicate with each other through a network (N).

[0035] The network (N) can perform wireless or wired communication between multiple user terminals (110-1, ���), a server (120), and a database (130). For example, the network can support wireless communication in the form of LTE (long-term evolution), LTE-A (LTE Advanced), CDMA (code division multiple access), WCDMA (wideband CDMA), WiBro (Wireless BroadBand), WiFi (wireless fidelity), Bluetooth, NFC (near field communication), GPS (Global Positioning System), or GNSS (global navigation satellite system). Additionally, the network (N) can perform wired communication in the form of USB (universal serial bus), HDMI (high definition multimedia interface), RS-232 (recommended standard 232), or POTS (plain old telephone service).

[0036] The database (130) has the ability to store various data. The data stored in the database (130) is data acquired, processed, or used by at least one component of a multiple user terminal (110-1, ���) or server (120), and may include software (e.g., programs). The database (130) may include volatile and / or non-volatile memory.

[0038] FIG. 2 is a diagram illustrating the learning of a neural network according to one embodiment.

[0039] As shown in FIG. 2, the learning device can train a neural network (123) to process review responses received from multiple user terminals (110-1, ���) item by item. Additionally, the learning device can train a neural network (123) to extract user stay records based on user movement path information. According to one embodiment, the learning device may be a separate entity from the server (120), but this is not limited to this.

[0040] The neural network (123) includes an input layer (121) and an output layer (125), where training samples are input and training outputs are generated. The network is trained based on the difference between the training outputs and the labels. The labels are defined based on items related to review responses and may also be defined based on user dwell records according to movement path information. The neural network (123) is connected in groups of multiple nodes and is defined by weights and activation functions between the nodes.

[0041] The learning device can train a neural network (123) using the Gradient Descent (GD) technique or the Stochastic Gradient Descent (SGD) technique. The device can utilize a loss function designed by the outputs and labels of the neural network. The learning device can calculate the training error using a predefined loss function. The loss function can predefine the labels, outputs, and parameters as input variables, and these parameters can be set as weights within the neural network (123). For example, the loss function can be designed in the form of Mean Square Error (MSE) or entropy, and various techniques can be applied.

[0042] The learning device can find weights that affect the training error using the backpropagation technique. Here, weights represent the relationships between nodes within the neural network (123). The learning device can use the SGD technique by utilizing labels and outputs to optimize the weights found by the backpropagation technique. For example, the learning device can update the weights of a loss function defined based on labels, outputs, and weights using the SGD technique.

[0043] According to one embodiment, the learning device extracts a first object from a review response, obtains a first label corresponding to the object, and applies it to a first neural network to generate a first training output corresponding to the first object. Subsequently, the first neural network can be trained based on the first training output and the first label. The learning device extracts a second object from movement path information, obtains a second label which is a user dwell record corresponding to the object, and applies it to a second neural network to generate a second training output corresponding to the second object. Subsequently, the second neural network can be trained based on the second training output and the second label.

[0044] According to one embodiment, the learning device can generate a first training feature vector based on the constituent features, position features, and pattern features of a review response, and various methods may be used for feature extraction. Additionally, the learning device can generate a second training feature vector based on the constituent features, length features, and pattern features of movement path information, and various methods may be used to extract features.

[0045] According to one embodiment, the learning device can obtain a training output by applying a first training feature vector to a neural network (123). Subsequently, the learning device can train a review item extraction algorithm of the neural network (123) based on the training output and the first label. The learning device can train the review item extraction algorithm by calculating a training error corresponding to the training output and optimizing the connection relationships between nodes within the neural network (123) to minimize this error. The server (120) can extract items from review responses by utilizing the first neural network that has completed training.

[0046] According to one embodiment, the learning device can obtain a training output by applying a second training feature vector to the neural network (123). Additionally, the learning device can train a user stay record acquisition algorithm of the neural network (123) based on the training output and the second label. The learning device can train the user stay record acquisition algorithm by calculating a training error corresponding to the training output and optimizing the connection relationships between nodes within the neural network (123) to minimize this error. The server (120) can obtain a user stay record from the movement path information through the second neural network after training is complete.

[0048] FIG. 3 is a diagram illustrating the configuration of an artificial intelligence model according to one embodiment.

[0049] According to one embodiment, an artificial intelligence model may be composed of an input layer, a hidden layer, and an output layer.

[0050] The input layer is the layer associated with the values ​​input into the artificial intelligence model. In the hidden layer, feature maps can be generated by performing multiply-accumulate (MAC) operations and activation operations on the input values. The MAC operation is a process of multiplying the input values ​​by their corresponding weights and then summing these multiplied values. The activation operation is a process of inputting the results of the MAC operation into an activation function to output a final result, and various types of activation functions can be used. For example, activation functions may include, but are not limited to, the sigmoid function, tangent function, ReLU function, Leaky ReLU function, Max Out function, and ELU function.

[0051] A hidden layer may consist of at least one layer, for example, divided into a first hidden layer and a second hidden layer. In this case, the first hidden layer generates a feature map by performing MAC operations and activation operations based on the input values ​​of the input layer, and this feature map can be used as the input value of the second hidden layer. The second hidden layer can perform MAC operations and activation operations based on the feature map generated as a result of the first hidden layer.

[0052] The output layer may be a layer related to the result of an operation performed in the hidden layer.

[0053] In one embodiment, a learning model learns syllable (character) patterns that are frequently used together in a given corpus to automatically identify the boundaries between compound words and named entities, and integrates object information from a first UI source with object information rendered in a browser to generate a learning object information file. Using this learning object information file, data for training a deep learning network is generated, and based on data received from various domains, the data is standardized into an integrated format according to one or more standardization methods suitable for each domain. Subsequently, data of a specific domain is trained and inferred, information necessary for standardization in that domain is determined, and post-processing can be performed on the data received from various domains.

[0054] The first UI source includes an XML file, and the training object information file includes an input JSON file for feature learning and an output JSON file used as label data during training. This output JSON file is a file containing HTML DOM Tree information implemented in compliance with web standards. Various domains include at least one of a RAN (radio access network), a transport, or a core, and the post-processing process may include a correlation function.

[0056] FIG. 4 is a block diagram showing the configuration of a panel management system according to one embodiment.

[0057] A system (400) according to one embodiment may include a database (402), a reliability evaluation module (404), a sampling module (406), a quality evaluation module (408), an administrator dashboard (410), a processor (420), and a memory (430), and some of the illustrated components may be omitted or substituted. A system (400) according to one embodiment may be a server or a terminal. The system (400) may operate in a cloud environment and may provide panel registration, survey responses, and community functions through a web or mobile client.

[0058] According to one embodiment, the processor (420) is configured to perform operations or data processing regarding the control and communication of each component of the system (400) and may be composed of one or more processors. The processor (420) may include at least one of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, or an application-specific integrated circuit (ASIC). The processor (420) can control the operation of the reliability evaluation module (404), the sampling module (406), and the quality evaluation module (408), and manage the data flow between each module.

[0059] According to one embodiment, memory (430) may store information related to the method described above or store a program in which the method described above is implemented. Memory (430) may be volatile memory (e.g., RAM) or non-volatile memory (e.g., ROM, flash memory, SSD). Memory (430) may store various file data, and the stored file data may be updated according to the operation of the processor (420). Memory (430) may store instructions including a panel confidence score calculation algorithm, a sampling algorithm, a response quality evaluation algorithm, and an attribute mismatch detection rule.

[0060] According to one embodiment, the processor (420) can execute a program and control the system (400). The code of the program executed by the processor (420) can be stored in memory (430). Operations of the processor (420) can be performed by loading instructions stored in memory (430). The system (400) can be connected to an external device (e.g., personal computer, mobile terminal, or network) through an input / output device (not shown in the drawing) and exchange data.

[0061] According to one embodiment, the database (402) may refer to a configuration that stores various data necessary for the operation of a panel management system. The database (402) may include a matching database, a member database, a panel database, and a response log database. The types of databases are merely examples and are not limited thereto, and additional databases may be included depending on the system design.

[0062] According to one embodiment, the matching database may refer to a database for mapping each user's nickname, internal unique number, and external authentication ID (e.g., social login ID, mobile phone authentication token). The matching database may serve as a key for consistently identifying the same user across multiple services, including panel management, community, and reward systems.

[0063] According to one embodiment, a member database may refer to a database that stores the panel's registration information and basic attribute information. The member database may store name, gender, date of birth, contact information, registration path, and consent information (e.g., consent to collection and use of personal information, consent to provision to third parties, consent to profiling). The personally identifiable information stored in the member database may be stored in an encrypted format, and only internal IDs may be used for panel analysis and survey analysis.

[0064] According to one embodiment, a panel database may refer to a database that integrates and stores the core attributes, detailed attributes, response history summaries, and behavioral indicators of a panel. Core attributes may include gender, age group, place of residence, education level, income level, occupation group, and political orientation. Detailed attributes may include marital status, presence of children, housing type, and pet ownership status and type. Response history summaries may include participation in each survey, participation time, response quality score, and frequency by topic category. Behavioral indicators may include average response speed, completion rate, linear response rate, and the number of logical inconsistency detections. The panel database may store the panel reliability score and reliability grade of each panel.

[0065] According to one embodiment, the response log database may refer to a database that stores response logs and quality evaluation results for each survey. The response log database may store survey IDs, question IDs, response content, response time, device information, and quality evaluation results in the form of raw logs. Partitioning or archiving may be applied to the response log database for analysis performance. Partitioning may refer to a method of dividing and storing large tables based on date, survey ID, or panel ID. Archiving may refer to a method of moving data that has elapsed for a certain period (e.g., one year, two years) to a separate storage to maintain the query and analysis performance of the current database.

[0066] According to one embodiment, the reliability evaluation module (404) can calculate a panel reliability score and identify attribute discrepancies based on attribute data, response history data, and behavioral data stored in the database (402). The reliability evaluation module (404) can operate under the control of the processor (420).

[0067] According to one embodiment, the reliability evaluation module (404) can calculate a panel reliability score by weighted summing the temporal consistency of attribute data, the frequency of participation per survey and response quality score history included in the response history data, and the average response speed, completion rate, and linear response rate included in the behavioral data. Based on the calculated panel reliability score, the reliability evaluation module (404) can classify each panel into one of a plurality of reliability grades.

[0068] According to one embodiment, the reliability evaluation module (404) can determine whether a change in the value of attribute data within a predetermined period corresponds to a predefined impossible change rule. An impossible change rule may refer to a rule that defines a type of attribute value change that cannot occur in a normal life course within a predetermined period. If the reliability evaluation module (404) determines that the change corresponds to an impossible change rule, it may exclude the survey response from the analysis and lower the reliability score of the panel.

[0069] According to one embodiment, the sampling module (406) may mean a configuration that automatically extracts a sample using survey request conditions, panel reliability, response history, and community activity history. The sampling module (406) may operate under the control of the processor (420).

[0070] According to one embodiment, the sampling module (406) may receive target demographic conditions (e.g., gender, age group, region, education level, income, occupation group), quota structure, required number of valid samples, minimum panel reliability grade, survey frequency limit, and other survey conditions as survey request conditions. The sampling module (406) may search for candidate panels satisfying the input conditions in the panel database and assign weights by considering each panel's reliability score, recent participation history, and response quality.

[0071] According to one embodiment, when target conditions including whether a specific product is owned, experience using a specific service, whether a specific activity is performed, or whether a specific interest is owned are input, the sampling module (406) can search past survey response history and community activity history to extract a set of panels that have been pre-verified as meeting the target conditions. The sampling module (406) can secure a high-reliability target sample without distributing separate screening questions.

[0072] According to one embodiment, the quality evaluation module (408) may be configured to receive a survey response stream, calculate a response quality score, and reflect the result in a database (402). The quality evaluation module (408) may operate under the control of a processor (420).

[0073] According to one embodiment, the quality evaluation module (408) can evaluate the response time per item, selection patterns, logical consistency, and text quality in real time while the panel responds to the survey. The quality evaluation module (408) can record the response time per item and detect abnormal response times relative to the average response time of the entire panel participating in the same survey. The quality evaluation module (408) can analyze linear response patterns in which the same option is repeated consecutively more than a preset number of times in matrix items. The quality evaluation module (408) can check logical consistency by determining whether the response value for the first item and the response value for the second item within the same survey correspond to a preset mutually exclusive condition rule.

[0074] According to one embodiment, the quality evaluation module (408) can calculate a text quality score by analyzing the number of characters entered for the free response text, the semantic similarity between the question sentence and the response sentence, and whether meaningless strings are included. The quality evaluation module (408) can calculate a survey unit response quality score by synthesizing each analysis result and store the calculated response quality score in the response log database. The quality evaluation module (408) can update the panel reliability score in the panel database based on the response quality score.

[0075] According to one embodiment, the administrator dashboard (410) may refer to a configuration that visualizes panel distribution, sampling results, quality indicators, and warning notifications and provides them to the administrator. The administrator dashboard (410) may be implemented as a web-based user interface and may be implemented using web frontend technologies such as React. The administrator dashboard (410) may provide JSON Web Token (JWT)-based authentication and role-based authorization management.

[0076] According to one embodiment, the administrator dashboard (410) can visualize and display the difference between the number of panels by attribute, including gender, age, region, education level, and income, and national statistics in the form of a graph or table. The administrator dashboard (410) can display quality indicators, including the response consistency rate by survey, the removal rate of duplicate and dishonest responses, and the distribution of panel grades. The administrator dashboard (410) can update and display the sampling status in real time, including the number of invitations by survey, the expected number of completions, and the actual number of completions.

[0077] According to one embodiment, the administrator dashboard (410) may display warnings and notifications including detection of attribute mismatch, increase in abnormal response patterns in a specific survey, and warnings of insufficient target samples. The administrator dashboard (410) may support a rapid response by sending an email, SMS, or push notification to the administrator when a warning occurs.

[0079] FIG. 5 illustrates the overall operation flow of a panel management system according to one embodiment.

[0080] The operations described through FIG. 5 can be implemented based on instructions that can be stored in memory (e.g., memory (420) of FIG. 4). The order of each operation of FIG. 5 may be changed, some operations may be omitted, and some operations may be performed simultaneously.

[0081] In operation 510, the system (400) can collect and store attribute data through the panel candidate's subscription information and profiling survey.

[0082] According to one embodiment, the system (400) may receive basic registration information, including name, contact information, registration path, and consent information for the collection of personal information, when a panel candidate performs membership registration through a web or mobile client. The system (400) may store the received basic registration information in a member database and maintain security by storing the personal identification information in an encrypted format.

[0083] According to one embodiment, the system (400) may collect core attribute data including gender, age, place of residence, education level, income level, occupation group, and political orientation by providing a profiling survey to panel candidates after registration is completed. In addition to core attributes, the system (400) may additionally collect detailed attribute data including marital status, presence of children, housing type, and ownership and type of pets. The types of core attributes are merely examples and are not limited thereto, and may vary depending on the settings.

[0084] According to one embodiment, the system (400) can store the collected attribute data in a panel database by mapping it to a panel unique ID. The system (400) can verify the initial reliability of the attribute data by comparing external authentication service information, including mobile phone identity verification or social login, with gender, age, and residence information entered at the time of registration to check for a match.

[0085] According to one embodiment, the system (400) may maintain the latest status of attribute data by performing periodic re-verification on the key attributes of the panel. The key attributes may include, for example, gender, age, place of residence, education level, income level, occupation group, and political orientation. The types of key attributes are merely examples and are not limited thereto and may vary depending on the settings. The system (400) may send a re-verification survey to each panel according to a preset cycle, and the re-verification cycle may be set, for example, every six months.

[0086] According to one embodiment, the system (400) can verify whether there is a match regarding gender, age, and place of residence by comparing the information entered at the time of registration with external authentication service information. The external authentication service may include mobile phone identity verification or social login. The system (400) can compare the gender, date of birth, and place of residence information included in the authentication token or authentication information provided by the external authentication service with the attribute values ​​entered by the panel.

[0087] According to one embodiment, the system (400) can check consistency regarding education level, income level, and occupation group by comparing past survey response history with newly collected self-information. The system (400) can compare attribute values ​​collected from a re-verification survey with past response values ​​stored in a response log database and check whether a discrepancy occurs.

[0088] According to one embodiment, the system (400) can calculate a political orientation index by synthesizing political and social survey responses and activity history within the platform. The system (400) can quantify the political orientation index by analyzing responses selected by a panel in a political survey, and posts or comments written on political topics in a community bulletin board. The system (400) manages the trend of changes in the political orientation index over time and can perform additional verification procedures if a sudden change occurs.

[0089] In operation 520, the system (400) can collect response history data and behavioral data when participating in a survey and calculate a reliability score for each panel.

[0090] According to one embodiment, the system (400) may store response history data, including a survey ID, question ID, response content, response time, and device information, in a response log database while the panel participates in a survey. The system (400) may summarize the participation status, participation time, response quality score, and frequency by topic category for each survey and reflect them in the response history summary field of the panel database.

[0091] According to one embodiment, the system (400) may calculate behavioral indicators including the panel's average response speed, survey completion rate, linear response rate, and the number of logical inconsistencies detected, and store them in a panel database. The system (400) may also collect activity history, such as the panel writing posts or comments on community bulletin boards within the platform, as behavioral data.

[0092] According to one embodiment, the system (400) can calculate a panel reliability score by weighting the temporal consistency of attribute data, the frequency of participation per survey and response quality score history included in response history data, and the average response speed, completion rate, and linear response rate included in behavioral data. Based on the calculated panel reliability score, the system (400) can classify each panel into one of a plurality of reliability grades, and the reliability grades may include, for example, Platinum, Gold, Silver, White, and Black. The types and number of grades are merely examples and are not limited thereto and may vary depending on the settings.

[0093] In operation 530, the system (400) can receive investigation request conditions and automatically select a panel of subjects for investigation based on a reliability score.

[0094] According to one embodiment, the system (400) may receive survey request conditions from a survey requester, including target demographic conditions, a quota structure, a required valid sample size, a minimum panel confidence level, survey frequency limits, and other survey conditions. The target demographic conditions may include gender, age group, region, education level, income, and occupation group, and the quota structure may be composed of a combination of gender, age group, and region.

[0095] According to one embodiment, the system (400) can search for candidate panels that satisfy the survey request conditions entered in the panel database. The system (400) can assign weights to each of the searched candidate panels by considering their reliability score, recent participation history, and response quality.

[0096] According to one embodiment, the system (400) can select a final sample by performing random and weighted sampling on weighted candidate panels. The system (400) can prevent response fatigue and maintain data quality by excluding from sample selection panels that have recently participated in a survey on the same topic within a predetermined period.

[0097] In operation 540, the system (400) can calculate a response quality score and update the reliability score per panel based on it.

[0098] According to one embodiment, the system (400) can evaluate the response time, selection pattern, logical consistency, and text quality for each response in real time while collecting survey responses from a selected sample. The system (400) can calculate a survey unit response quality score by synthesizing the evaluated results.

[0099] According to one embodiment, the system (400) may store the calculated response quality score in the response log database and reflect it in the panel reliability score of the panel database. If the response quality score accumulates below a standard for a predetermined period, the system (400) may lower the reliability rating of the panel or switch it to a blacklist status.

[0100] According to one embodiment, the system (400) can continuously prioritize high-quality respondents by reflecting the updated panel reliability score in the subsequent sampling process. The system (400) can visualize the quality evaluation results on an administrator dashboard to provide quality indicators including the response consistency rate per survey, the removal rate of duplicate and dishonest responses, and the distribution of panel grades.

[0101] According to one embodiment, the system (400) may collect attribute data including gender, age, residence, education level, income level, occupation group, and political orientation through the panel candidate's registration information and profiling survey, and store this data in a panel database. The system (400) may collect response history data and behavioral data including response content, response time, and selection patterns while the panel candidate participates in the survey. The system (400) may calculate a reliability score for each panel based on the attribute data, response history data, and behavioral data. If the value of the attribute data changes within a predetermined period, the system (400) may detect attribute inconsistencies by determining whether the change is a combination that is not realistically possible according to predefined rules. The system (400) may receive target demographics, required sample size, and minimum reliability criteria as survey request conditions and automatically select a survey target panel as a sample using the reliability score for each panel. While collecting survey responses from the sample, the system (400) may calculate a response quality score by evaluating response time, selection patterns, logical consistency, and text quality. The system (400) can update the reliability score per panel based on the response quality score.

[0103] According to one embodiment, the system (400) may calculate a panel reliability score by weighting the sum of the temporal consistency of attribute data, the frequency of participation and response quality score history per survey included in response history data, and the average response speed, completion rate, and linear response rate included in behavioral data. Based on the calculated panel reliability score, the system (400) may classify each panel into one of a plurality of reliability grades. If the response quality score accumulates below a standard for a predetermined period, the system (400) may control the panel to lower its reliability grade or switch it to a blacklist status.

[0104] According to one embodiment, the panel reliability score may be a numerical value that quantitatively indicates how reliable an individual panel is as a survey respondent. The panel reliability score may be calculated by combining multiple evaluation factors rather than a single indicator, thereby enabling the distinction between panels that are excellent only in specific aspects and panels that are generally reliable.

[0105] According to one embodiment, temporal consistency of attribute data may refer to whether a panel has provided consistent responses to the same attribute item in multiple surveys conducted at different times. The system (400) may query past attribute response records stored in the panel database and calculate a temporal consistency score by analyzing whether the response values ​​for the same attribute have changed and the frequency of change. A panel whose attribute responses are relatively consistent may be assigned a relatively high temporal consistency score, while a panel whose attribute values ​​change relatively more frequently may be assigned a relatively low score.

[0106] According to one embodiment, the system (400) may calculate a temporal consistency score by counting the number of attribute data changes during a preset reference period (e.g., 6 months or 1 year). For example, the system (400) may assign a high temporal consistency score with maximum weighting to panels in which the number of changes to key attribute data, including occupation, residence, or income level, within the reference period is 1 or less than a first threshold (e.g., 1 time). On the other hand, if the number of changes within the same period exceeds a second threshold (e.g., 3 times), the system (400) may consider this as abnormal data fluctuation and assign a low score below the threshold or reflect it as a deduction factor when calculating reliability. The number of changes and the threshold values ​​are merely examples and are not limited thereto, and may vary depending on the settings.

[0107] According to one embodiment, the frequency of participation per survey may refer to the number of surveys in which a panel participates during a certain period. The system (400) may aggregate the number of survey participations per panel by querying a response history database. A panel that maintains an appropriate level of participation frequency may be evaluated as a stable respondent, while an excessively low participation frequency may be evaluated as a lack of data accumulation, and an excessively high participation frequency may be evaluated as a possibility of quality degradation due to response fatigue.

[0108] According to one embodiment, the response quality score history may refer to a cumulative record of response quality scores obtained in each survey in which the panel participated in the past. The system (400) can retrieve the response quality scores for each survey stored in the response log database and analyze them in a time series to identify the response quality trends of the panel. A panel that consistently maintains high response quality scores may be evaluated positively in the calculation of the reliability score.

[0109] According to one embodiment, the average response speed may refer to the average value of the time taken for a panel to respond to a question. The system (400) may query the response time per question recorded in the response log database and calculate the average value for each panel. A panel that maintains an average response speed within an appropriate range may be evaluated as having read the question sufficiently and responded, while an excessively fast average response speed may indicate the possibility of a half-hearted response.

[0110] According to one embodiment, the completion rate may refer to the percentage of panelists who participate in a survey and complete the response to the end. The system (400) may calculate the percentage of surveys completed without dropping out among the surveys in which the panelists participated. Panelists who maintain a high completion rate may be evaluated as conscientious respondents, while panelists who frequently drop out may be assigned a low completion rate score.

[0111] According to one embodiment, the linear response rate may refer to the rate at which a panelist repeatedly selects the same option in a matrix question. The system (400) can calculate the frequency of linear responses by analyzing response patterns for matrix questions in a response log database. Panelists with a high linear response rate may be evaluated negatively in the calculation of reliability scores because they may have responded carelessly.

[0112] According to one embodiment, the system (400) may assign weights to each of the evaluation elements and calculate a panel reliability score through a weighted summation method. The weights may be set differently depending on the purpose of the survey or the system operation policy. For example, in a survey where the accuracy of attribute data is important, a high weight may be assigned to temporal consistency, and in a survey where response sincerity is important, a high weight may be assigned to the linear response rate or the completion rate.

[0113] According to one embodiment, the system (400) can map the calculated panel reliability score to one of a plurality of reliability grades according to a preset score range. The reliability grade may affect priority and eligibility for participation during sampling. Panels with a high grade may be preferentially selected for high-quality surveys, while panels with a low grade may be excluded from sampling or participate only on a limited basis.

[0114] According to one embodiment, the system (400) may automatically lower the reliability rating of a panel if the response quality score remains below a preset threshold for a predetermined period or accumulates. The predetermined period may be set, for example, in units of one month or three months, and the threshold score and period may be adjusted by the system administrator. When the rating is lowered, the system (400) may send a notification to the panel and provide an opportunity for improvement.

[0115] According to one embodiment, the system (400) may convert panels whose reliability rating has been lowered to the lowest grade or panels that consistently fail to meet quality standards into a blacklisted state. Panels in a blacklisted state may be automatically excluded from all subsequent sampling processes, and this may have the effect of systematically excluding respondents who impair the quality of survey data. The system (400) may record the reason for the blacklist conversion and the date in the panel database to be used for future review or restoration decisions.

[0116] According to one embodiment, the system (400) can establish a coalition quality network that shares panel quality information among a plurality of panel management systems. A coalition quality network may refer to a network in which panel management systems operated by different survey companies or survey platforms mutually share information about malicious panels.

[0117] According to one embodiment, the system (400) may share hashed panel identifiers and quality summary indicators with other panel management systems participating in the federated quality network. A hashed panel identifier may refer to an identifier generated by applying a cryptographic hash function to identify the same panel without directly exposing the panel's personal identification information. The quality summary indicators may include information summarizing the panel's trust rating, whether it has been blacklisted, and the history of major discrepancies detected.

[0118] According to one embodiment, the system (400) can query quality summary indicators from a federated quality network when sampling. Based on the queried quality summary indicators, the system (400) can exclude panels designated as blacklisted in other panel management systems from sample selection. Through this, the system (400) can exclude in advance panels that have not yet been detected in its own system but have already been identified as malicious in other systems.

[0119] According to one embodiment, the federated quality network may be operated in a manner that shares only hash values, rather than the actual identification information of the panel, for the protection of personal information. The system (400) may apply the same hash function to the panel identifier stored in its own panel database and compare it with the hash value retrieved from the federated network. Since the hash function is a one-way function and it is impossible to reverse-calculate the original identification information from the hash value, the exposure of the panel's personal information can be prevented.

[0120] According to one embodiment, an online survey sample extraction, quality evaluation, and result prediction method using a panel management system may include: collecting attribute data including gender, age, residence, education level, income level, occupation group, and political orientation through the registration information and profiling survey of panel candidates and storing it in a panel database; collecting response history data and behavioral data including response content, response time, and selection pattern while panel candidates participate in the survey; calculating a reliability score per panel based on the attribute data, response history data, and behavioral data; detecting attribute inconsistencies by determining whether the change is a combination that is not realistically possible according to predefined rules when the value of the attribute data changes within a predetermined period; receiving target demographics, required sample size, and minimum reliability criteria as survey request conditions and automatically selecting a survey target panel as a sample using the reliability score per panel; calculating a response quality score by evaluating response time, selection pattern, logical consistency, and text quality while collecting survey responses from the sample; and updating the reliability score per panel based on the response quality score.

[0121] According to one embodiment, an online survey sample extraction, quality evaluation, and result prediction method using a panel management system may include, in calculating a reliability score for each panel, an operation to calculate a panel reliability score by weighted summing the temporal consistency of attribute data, the frequency of participation and response quality score history for each survey included in response history data, and the average response speed, completion rate, and linear response ratio included in behavioral data; an operation to classify each panel into one of a plurality of reliability grades based on the calculated panel reliability score; and an operation to control the reliability grade of the panel to be lowered or switched to a blacklist state if the response quality score accumulates below a standard for a predetermined period.

[0123] FIG. 6 illustrates the operation flow of attribute mismatch detection in a panel management system according to one embodiment.

[0124] The operations described through FIG. 6 can be implemented based on instructions that can be stored in memory (e.g., memory (420) of FIG. 4). The order of each operation of FIG. 6 may be changed, some operations may be omitted, and some operations may be performed simultaneously.

[0125] In operation 610, the system (400) can compare the attribute value that the same panel responded to in the first survey with the attribute value that the same panel responded to in the second survey.

[0126] According to one embodiment, the system (400) can retrieve past survey response history stored in the panel database and the response log database to extract attribute response records of a panel identified by the same panel identifier. The system (400) can calculate the time interval between the time of response to the first survey and the time of response to the second survey from the extracted attribute response records.

[0127] According to one embodiment, the system (400) can compare attribute values ​​answered in a first survey and attribute values ​​answered in a second survey performed within a predetermined period for key attributes including educational background, income level, and occupation group. The predetermined period may be set, for example, in units of one week or one month, and may be adjusted by a system administrator.

[0128] In operation 620, the system (400) can determine whether the attribute value change corresponds to an impossible change rule.

[0129] According to one embodiment, the system (400) may determine whether a change in an attribute value corresponds to a type of change that cannot occur in a normal life course within a predetermined period by referring to a database of predefined impossible change rules. The impossible change rules may include, for example, cases where the highest level of education changes from high school graduation to graduate school graduation within one week, or cases where the income level changes drastically within a short period.

[0130] According to one embodiment, the system (400) may apply different impossible change rules for each attribute type. For example, in the case of an education level attribute, since a minimum period of time is required to change to a higher education level, it may be determined that a rapid upward change within a short period of time is impossible.

[0131] In operation 630, the system (400) may exclude the response and lower the reliability score when it determines that the change is impossible.

[0132] According to one embodiment, if the system (400) determines that a survey response corresponds to an impossible change rule, it may exclude the corresponding response from analysis and record the reason for exclusion in the response log database. The system (400) may lower the reliability score of the panel by a pre-set deduction factor and reflect it in the panel database.

[0133] According to one embodiment, the system (400) may automatically register a panel for which an impossible change has been detected as a subject for a re-authentication survey or add it to an operator review queue. The system (400) may reconfirm the actual attribute values ​​of the panel through the re-authentication survey and update the attribute information in the panel database according to the confirmed results.

[0134] In operation 640, the system (400) may switch the panel to an unreliable state when the discrepancy accumulates more than a predetermined number of times.

[0135] According to one embodiment, the system (400) may switch a panel to an untrusted state if attribute mismatches corresponding to an impossible change rule for the same panel accumulate more than a predetermined number of times. Here, the predetermined number of times may be set by an administrator, for example, to three times or more.

[0136] According to one embodiment, the system (400) may classify panels that have been switched to an untrusted state into a blacklist grade and automatically exclude them during the subsequent sampling process. The system (400) may display information about panels classified as blacklisted on an administrator dashboard so that an operator can check the status of the panel and manually change the status if necessary. Through this attribute mismatch detection function, the system (400) can effectively exclude panels that repeat false statements and maintain the overall quality of the panel pool.

[0137] According to one embodiment, in detecting attribute discrepancies, the system (400) may compare the attribute value responded to in a first survey by a panel identified by the same panel identifier with the attribute value responded to in a second survey performed within a predetermined period. The system (400) may determine whether the change in the attribute value resulting from the comparison corresponds to a predefined impossible change rule, which is a type of change that cannot occur in a normal life course within a predetermined period. If it is determined that it corresponds to an impossible change rule, the system (400) may exclude the survey response from the analysis target and lower the reliability score of the panel by a pre-set deduction coefficient. If attribute discrepancies corresponding to the impossible change rule accumulate more than a predetermined number of times, the system (400) may switch the panel to an unreliable state and automatically exclude it from the subsequent sampling process.

[0139] According to one embodiment, the system (400) may receive target conditions as survey request conditions when automatically selecting a survey target panel as a sample, including at least one of the following: possession of a specific product, experience using a specific service, performance of a specific activity, or possession of a specific interest. The system (400) may search past survey response history stored in a panel database and extract a first candidate group consisting of panels that have consistently provided responses matching the target conditions on multiple occasions. The system (400) may search a community activity history database and extract a second candidate group consisting of panels that have a history of writing posts on topics related to the target conditions or writing comments on posts on such topics. By integrating the first candidate group and the second candidate group to create a survey target candidate group, the system (400) can secure a set of panels that have been pre-verified as meeting the target conditions without distributing separate screening questions. The system (400) may control the selection of a final sample from among panels that meet or exceed the minimum reliability standard by querying the reliability score of each panel included in the survey target candidate group.

[0140] According to one embodiment, target conditions may refer to specific eligibility requirements set by a research client to select respondents who meet the objectives of a specific survey. Target conditions may be established based on specific experiences, ownership status, or interests that are difficult to identify solely by demographic attributes. For example, ownership of a specific product may include whether the individual currently owns a large dog, an electric vehicle, or home appliances of a specific brand. Experience using a specific service may include whether the individual has used online streaming services, subscription-based food delivery services, or specific financial products. Performance of a specific activity may include whether the individual has a history of participating in a marathon, traveling abroad, or volunteering. Possession of specific interests may include whether the individual is interested in specific topics such as eco-friendly products, investment, or healthcare. The types of target conditions are merely examples and are not limited thereto; they may be set in various ways depending on the purpose of the survey.

[0141] According to one embodiment, the system (400) may provide an interface for receiving target conditions from a research requester. The interface may be configured to allow selecting a type of target condition and inputting detailed conditions in the form of text or options. The system (400) may convert the input target conditions into the form of keywords or conditional expressions and utilize them for database searching.

[0142] According to one embodiment, the system (400) can identify responses that meet target conditions by searching past survey response history stored in the panel database and the response log database. Searching past survey response history may mean looking up how a panel responded to questions related to target conditions in surveys in which they previously participated. For example, in the case of a survey targeting owners of large dogs, the system (400) can search for a panel that has a record of responding that they own a large dog in response to a question asking about whether they own a pet and the type of pet in a past survey.

[0143] According to one embodiment, the system (400) may extract a panel that has consistently provided a response meeting the target condition over multiple occasions, rather than just once, as a first candidate group. Consistent responses over multiple occasions may mean consistently providing a positive response to the same or similar target-related questions in multiple surveys conducted at different times. The system (400) may include a panel whose number of consistent responses exceeds a preset standard in the first candidate group. Through this method, the system (400) can select a panel that is confirmed to continuously meet the target condition, rather than providing a temporary or accidental response.

[0144] According to one embodiment, the community activity history database may refer to a database that stores records of activities performed by a panel on a community bulletin board within the platform. The community activity history may include the title and body of a post written by the panel, comment content, the date and time of the activity, and the activity bulletin board category. The system (400) can search the community activity history database using keywords related to target conditions to identify a panel that has actually taken an interest in the topic and engaged in activities.

[0145] According to one embodiment, the system (400) may extract a second group of candidates consisting of panels that have a history of writing posts on topics related to target conditions or writing comments on posts on said topics. For example, in the case of a survey targeting owners of large dogs, the system (400) may search for panels that have written posts on topics such as experience raising large dogs, recommendations for large dog supplies, and health management for large dogs, or have written comments on related posts on a pet community bulletin board. By utilizing community activity history, the system (400) can indirectly verify the actual interests and experiences of the panels that are difficult to ascertain solely from survey responses.

[0146] According to one embodiment, the system (400) can generate a candidate group for investigation by integrating a first candidate group and a second candidate group. The integration method may be set to either a union method or an intersection method. The union method may be a method of including all panels included in either the first candidate group or the second candidate group in the candidate group for investigation. The intersection method may be a method of including only panels included in both the first candidate group and the second candidate group in the candidate group for investigation. The system (400) may select the integration method according to the requirements of the survey requester or the required number of samples.

[0147] According to one embodiment, the system (400) can secure a pre-verified set of panels that meet target conditions without distributing separate screening questions by generating a candidate group of survey subjects. A screening question may refer to a question inserted at the beginning of a survey to verify whether the respondent is qualified to be a survey subject. In conventional methods, screening questions were distributed to a large number of respondents to secure a valid sample, and the main survey was conducted only on respondents who met the qualification requirements; however, this process increased time and costs, and problems could arise where unqualified respondents gave false positive responses. By utilizing a pre-verified candidate group based on past response history and community activity history, the system (400) can reduce costs associated with distributing screening questions and prevent the influx of false respondents.

[0148] According to one embodiment, the system (400) can retrieve the reliability score of each panel included in the survey candidate group from the panel database. The system (400) can compare the retrieved reliability score with the minimum reliability criterion included in the survey request conditions and filter only the panels that are equal to or greater than the minimum reliability criterion as the final sample selection targets. Through this, the system (400) can select only the panels that satisfy the target conditions and have verified response reliability as the final sample.

[0149] According to one embodiment, the system (400) can select a final sample by considering the required number of valid samples and the quota structure for the filtered panel. The system (400) can ensure the representativeness of the sample by applying a random sampling or weighted sampling method. Through this automatic sampling process, the system (400) can help the survey client obtain high-quality samples that meet target conditions without any separate manual work.

[0150] According to one embodiment, the system (400) may include a minimum number of screening questions in the survey as needed and detect false respondents by cross-validating the screening responses with past history. Cross-validation may mean verifying consistency by comparing the current response to the screening question with the past response history stored in the panel database.

[0151] According to one embodiment, the system (400) can detect a discrepancy if a panelist who responded in a past response history that they do not own a pet responds in a current screening question that they own a large dog. The system (400) can classify the response in which the discrepancy is detected as a false response and exclude the panelist from the main survey. Through this cross-verification, the system (400) can exclude respondents who intend to falsely meet eligibility requirements solely for the purpose of obtaining a reward.

[0153] FIG. 7 is a flowchart illustrating the operation of calculating a response quality score of a panel management system according to one embodiment.

[0154] The operations described through FIG. 7 can be implemented based on instructions that can be stored in memory (e.g., memory (420) of FIG. 4). The order of each operation of FIG. 7 may be changed, some operations may be omitted, and some operations may be performed simultaneously.

[0155] In operation 710, the system (400) can record the response time per item and detect abnormal response times relative to the average.

[0156] According to one embodiment, the system (400) can record the time taken by the panel to respond to each question in milliseconds and store it in a response log database. The system (400) can also aggregate response times in page units to calculate the time taken per question group.

[0157] According to one embodiment, the system (400) can calculate the average response time per question for all panel members who participated in the same survey. The system (400) can detect an abnormal response time if the response time of a specific panel member is below a preset lower limit ratio or above a preset upper limit ratio relative to the average response time. For example, the lower limit ratio may be set to 20% or less of the average, and the upper limit ratio may be set to 500% or more of the average. The magnitudes of the lower and upper limit ratios are merely examples and may vary depending on the settings.

[0158] According to one embodiment, the system (400) can calculate a response time quality score by aggregating the number of items in which abnormal response times were detected. The system (400) can use a short response time based on a specified time as an indicator of a half-hearted response, and a response time longer than a certain level based on a specified time as an indicator of disengagement or distraction.

[0159] In operation 720, the system (400) can analyze a linear response pattern in which the same option is repeated in the matrix items.

[0160] According to one embodiment, the system (400) can extract a sequence of option numbers selected by a panel in a matrix question containing a plurality of sub-questions. The system (400) can calculate the number of times the same option number appears consecutively in the extracted sequence.

[0161] According to one embodiment, the system (400) can determine that the same option number is repeated consecutively more than a preset number of times as a linear response pattern. The preset number can be set to, for example, 5 times or more and can be adjusted by a system manager.

[0162] According to one embodiment, the system (400) can calculate a selection pattern quality score by aggregating the number of matrix items in which a linear response pattern is detected. In addition to linear responses, the system (400) can comprehensively determine whether the response is insincere by further analyzing regular selection patterns including periodic patterns or diagonal patterns.

[0163] In operation 730, the system (400) can check whether the responses to the first and second questions correspond to mutually exclusive conditions.

[0164] According to one embodiment, the system (400) can extract response values ​​for a first question and a second question that are logically related within the same survey. The system (400) can determine whether the combination of response values ​​for the first question and response values ​​for the second question is logically contradictory by referring to a predefined mutually exclusive condition rule database.

[0165] According to one embodiment, mutually exclusive condition rules may include, for example, cases where a panelist who responded that their monthly income is 0 won responds that they regularly purchase expensive products in the same survey, or cases where a panelist who responded that they have no children responds about spending on children's education expenses.

[0166] According to one embodiment, the system (400) can calculate a logical consistency score by aggregating the number of times response combinations corresponding to mutually exclusive conditions are detected. The system (400) evaluates the reliability of the corresponding response as low for panels in which logical inconsistencies are detected, and can use repeated logical inconsistencies as grounds for lowering the reliability score.

[0167] In operation 740, the system (400) can analyze the number of characters, semantic similarity, and whether the free response text contains meaningless strings.

[0168] According to one embodiment, the system (400) can calculate the number of characters of free response text entered into a subjective question and determine whether the number of characters is greater than or equal to a preset minimum number of characters. The minimum number of characters may be set differently depending on the type of question.

[0169] According to one embodiment, the system (400) can calculate semantic similarity between a question sentence and an answer sentence using natural language processing technology. The system (400) can evaluate whether the response is relevant to the question by determining whether the calculated semantic similarity is greater than or equal to a preset threshold.

[0170] According to one embodiment, the system (400) can analyze whether the response text contains meaningless strings, repetitive strings, profanity, or suspected auto-generated patterns. The system (400) can detect inappropriate text through keyword database comparison and text pattern analysis, and reflect the detection results in a text quality score.

[0171] In operation 750, the system (400) can synthesize each analysis result to calculate a survey unit response quality score.

[0172] According to one embodiment, the system (400) can calculate a survey unit response quality score by weighted summing the response time quality score calculated in operation 710, the selection pattern quality score calculated in operation 720, the logical consistency score calculated in operation 730, and the text quality score calculated in operation 740. The weights applied to each score may be set differently depending on the survey type or the purpose of the survey.

[0173] According to one embodiment, the system (400) may store the calculated survey unit response quality score in a response log database and reflect the score in the update of the panel confidence score. The system (400) may exclude responses with a response quality score below a standard from the analysis or classify them as separate review subjects.

[0174] According to one embodiment, the system (400) can process quality evaluation results in real time to detect and respond to abnormal responses even while the survey is in progress. The system (400) can visualize the quality evaluation results on an administrator dashboard so that an operator can check the status of quality indicators for each survey.

[0175] According to one embodiment, the system (400) can calculate a response quality score by recording the response time per question and per page, calculating the average response time per question for all panelists participating in the same survey, and detecting as an abnormal response time if the response time of a specific panelist is below a preset lower limit ratio or above a preset upper limit ratio compared to the average response time. The system (400) can determine whether a response is insincere by analyzing a linear response pattern in which the same option number is repeated consecutively more than a preset number of times in a matrix question. The system (400) can check logical consistency by determining whether the response value for the first question and the response value for the second question within the same survey correspond to a preset mutually exclusive condition rule. The system (400) can calculate a text quality score by analyzing whether the number of characters entered for the free response text is greater than or equal to a preset minimum number of characters, whether the semantic similarity between the question sentence and the response sentence is greater than or equal to a preset threshold, and whether meaningless strings or repetitive strings are included. The system (400) can control the calculation of a survey unit response quality score by combining the abnormal response time detection result, the linear response pattern determination result, the logical consistency check result, and the text quality score.

[0176] According to one embodiment, the system (400) may provide an administrator dashboard that visualizes panel distribution, sampling results, quality indicators, and warning notifications. The administrator dashboard may be implemented as a web-based user interface and may support JSON Web Token (JWT)-based authentication and role-based authorization management.

[0177] According to one embodiment, the system (400) can visualize and display the difference between the number of panels by attribute, including gender, age, region, education level, and income, and national statistics through an administrator dashboard. The system (400) can provide the difference between the actual panel distribution and the target population distribution in the form of a graph or table to evaluate the representativeness of the panel pool.

[0178] According to one embodiment, the system (400) can display quality indicators including the response consistency rate per survey, the removal rate of duplicate and dishonest responses, and the panel grade distribution through an administrator dashboard. The system (400) can update and display the sampling status in real time, including the number of invitations per survey, the expected number of completions, and the actual number of completions.

[0179] According to one embodiment, the system (400) can display warnings and notifications through an administrator dashboard, including detection of attribute mismatches, an increase in abnormal response patterns in a specific survey, and warnings of insufficient target samples. The system (400) can support a rapid response by sending an email or push notification to the administrator when a warning occurs.

[0180] According to one embodiment, the system (400) can generate a digital twin profile in the form of a multidimensional vector that quantifies the response propensity of each panel by using attribute data, topic-specific response history data, response quality score history, and behavioral data of each panel as inputs. The digital twin profile may include response propensity parameters that include the response distribution by topic category of surveys the panel participated in in the past, preference choice trends, positive or negative response ratios, and price sensitivity indices. When a new survey is planned, the system (400) can generate a survey characteristic vector by analyzing the question structure, topic category, choice type, and response scale of the new survey. The system (400) can simulate what response distribution each digital twin profile will exhibit for the new survey by calculating the similarity between the survey characteristic vector and a plurality of digital twin profiles. The system (400) can improve prediction accuracy by periodically updating the weight parameters of the prediction model using the error between the actual response results of similar surveys conducted in the past and the prediction results based on the digital twin profile at that time as training data. The system (400) can be controlled to generate a survey result prediction report including at least one of an expected preference distribution, an expected purchase intention rate, or an expected conversion rate for a new survey based on a simulated response distribution.

[0181] According to one embodiment, a digital twin profile may refer to a virtual profile modeled based on data regarding the response tendencies and behavioral characteristics of an actual panel. The digital twin profile can be used as basic data to predict how an individual panel will respond to a specific type of survey. The system (400) can generate and maintain a digital twin profile corresponding to each panel based on data accumulated in the panel database and the response log database.

[0182] According to one embodiment, the system (400) may use attribute data, topic-specific response history data, response quality score history, and behavioral data as inputs to generate a digital twin profile. The attribute data represents the demographic characteristics of the panel, and the topic-specific response history data may include the topic categories of surveys the panel has previously participated in and the content of the responses in those surveys. The response quality score history represents the level of response sincerity of the panel, and the behavioral data may include the average response speed, completion rate, and selection patterns.

[0183] According to one embodiment, the system (400) can process the input data to generate a digital twin profile in the form of a multidimensional vector. A multidimensional vector refers to a data structure composed of multiple numerical values, and each dimension may represent a specific response tendency or characteristic of the panel. The system (400) can perform preprocessing on the input data, including normalization, numerical conversion of categorical variables, and processing of missing values, and then convert it into a vector form.

[0184] According to one embodiment, the response propensity parameter included in the digital twin profile may refer to a set of indicators that quantitatively represent the response characteristics of the panel. The response distribution by topic category may refer to the frequency distribution of selecting a specific type of response in each topic, after classifying the surveys in which the panel participated in the past by topic. For example, the system (400) may construct the response distribution by topic by calculating the proportion of positive responses selected by the panel in a consumer satisfaction survey or the proportion of specific positions selected in a political orientation survey.

[0185] According to one embodiment, a preference choice tendency may refer to a tendency of a panel to prefer a choice at a specific position among a plurality of choices. The system (400) may calculate the preference choice tendency by analyzing the frequency with which the panel selects extreme values, the frequency with which it selects middle values, or the frequency with which it selects the first choice in Likert scale items.

[0186] According to one embodiment, the positive or negative response ratio may refer to the ratio of panelists selecting positive and negative responses in a question asking about satisfaction, preference, or agreement. The system (400) can determine whether the panelists' overall response tendency is positive or negative through this ratio.

[0187] According to one embodiment, the price sensitivity index may refer to an indicator that quantifies how sensitively a panel responds to price changes in price-related questions. The system (400) can calculate the price sensitivity index by analyzing the response patterns selected by the panel in questions asking about value relative to price, willingness to pay, or changes in willingness to purchase upon price increase. The types of response propensity parameters are merely examples and are not limited thereto, and various parameters may be added depending on the purpose of the survey.

[0188] According to one embodiment, when a new survey is planned, the system (400) can generate a survey characteristic vector by analyzing the structural characteristics of the survey. A survey characteristic vector may refer to a numerical representation of the content and structural characteristics of a survey in the form of a vector. The system (400) can construct the survey characteristic vector by analyzing the number of items included in the survey, the distribution of item types, topic categories, the types of response scales used, and the number of options.

[0189] According to one embodiment, the system (400) can calculate similarity between a survey feature vector and a plurality of digital twin profiles. Calculating similarity may mean quantifying the degree of similarity by calculating the distance or correlation between two vectors. The system (400) may use any one of similarity measurement methods including cosine similarity, Euclidean distance, or Pearson correlation coefficient. The system (400) can infer an expected response distribution for a new survey by referring to response patterns from past surveys that have high similarity.

[0190] According to one embodiment, the system (400) can simulate what kind of response distribution each digital twin profile will show for a new survey based on the similarity calculation results. Simulation may refer to the operation of generating virtual response results by inputting response propensity parameters and survey characteristic vectors included in the digital twin profile into a prediction model. The system (400) can perform simulations on a plurality of digital twin profiles and aggregate the results to calculate the expected response distribution for the entire sample.

[0191] According to one embodiment, the system (400) may use the error between the actual response results of similar surveys conducted in the past and the prediction results based on the digital twin profile at that time as training data to improve the accuracy of the prediction model. The system (400) may define the difference between the prediction result and the actual result as a loss function and adjust the weight parameters of the prediction model through gradient descent or a similar optimization algorithm. The system (400) may improve the prediction accuracy by updating the prediction model whenever new survey results are collected or periodically.

[0192] According to one embodiment, the system (400) may generate a survey result prediction report based on a simulated response distribution. A survey result prediction report may refer to a document or data that summarizes and provides expected response results before a new survey is actually distributed. The prediction report may include an expected preference distribution, an expected purchase intention ratio, or an expected conversion rate. An expected preference distribution may refer to a distribution of the proportion of preference responses for a specific product or service. An expected purchase intention ratio may refer to the proportion predicted to respond positively to a question asking about purchase intention. An expected conversion rate may refer to the proportion predicted to indicate an intention to convert to a specific action.

[0193] According to one embodiment, the system (400) provides a survey result prediction report to the survey requester, which can be used as decision support information to determine whether to actually distribute the survey, the number of samples required, and the priority of target segments. The survey requester may refer to the prediction report to skip the actual survey if the expected results are already clear, or increase the number of samples for a specific segment if the expected results are uncertain in that segment.

[0195] FIG. 8 is a block diagram showing the database structure of a panel management system according to one embodiment.

[0196] According to one embodiment, the database (800) may refer to a configuration that stores various data necessary for the operation of a panel management system. The database (800) may include a matching DB (802), a member DB (804), a panel DB (806), and a response log DB (808). The types of individual databases included in the database (800) are merely examples and are not limited thereto; additional databases may be included or some may be integrated depending on the system design.

[0197] According to one embodiment, the matching DB (802) may refer to a database for mapping each user's nickname, internal unique number, and external authentication ID. The external authentication ID may include a social login ID or a mobile phone authentication token. The matching DB (802) may serve as a key for consistently identifying the same user across multiple services, including panel management, community, and reward systems.

[0198] According to one embodiment, the matching DB (802) can support integrated management with a single internal unique number even when a user has linked multiple external authentication methods. The matching DB (802) can store the user's nickname change history to maintain continuity with past activity records. The matching DB (802) can manage token status information so that it can detect when the token of an external authentication service is renewed or expires and request re-authentication from the user.

[0199] According to one embodiment, the member DB (804) may refer to a database that stores the panel's registration information and basic personal information. The member DB (804) may store name, gender, date of birth, contact information (e.g., mobile phone number, email address), registration method (e.g., direct website registration, mobile app registration, affiliate linkage registration), and consent information. The consent information may include consent for the collection and use of personal information, consent for provision to third parties, and consent for profiling.

[0200] According to one embodiment, personally identifiable information (PII) stored in the member DB (804) may be stored in an encrypted form. The encryption method may be symmetric key encryption such as AES-256 or asymmetric key encryption such as RSA. The member DB (804) may be linked with other databases using only internal IDs so that personally identifiable information is not directly exposed during panel analysis and survey analysis processes.

[0201] According to one embodiment, the panel DB (806) may refer to a database that integrates and stores attribute information, response history summaries, and behavioral indicators of a panel. The panel DB (806) may assign a unique panel ID to each panel and store various information based on this.

[0202] According to one embodiment, the panel DB (806) may store core attributes. Core attributes may include, for example, gender, age group, place of residence, education level, income level, occupation group, and political orientation. The panel DB (806) may store detailed attributes. Detailed attributes may include, for example, marital status, presence of children, housing type (e.g., homeownership, Jeonse, monthly rent), pet ownership status and type (e.g., large dog, small dog, cat).

[0203] According to one embodiment, the panel DB (806) may store a response history summary. The response history summary may include participation status per survey, time of participation, response quality score, and frequency of participation by topic category. The panel DB (806) may store behavioral indicators. The behavioral indicators may include average response speed, survey completion rate, linear response rate, and number of logical inconsistencies detected.

[0204] According to one embodiment, the panel DB (806) may store the panel reliability score (PRScore) and reliability grade of each panel. The reliability grade may include, for example, Platinum, Gold, Silver, White, and Black. The types and names of the reliability grades are merely examples and are not limited thereto, and may vary depending on the system operation policy.

[0205] According to one embodiment, the response log DB (808) may refer to a database that stores response logs and quality evaluation results for each survey. The response log DB (808) may store a survey ID, question ID, response content, response time (e.g., time taken per question, time taken per page), device information (e.g., type of web browser, operating system, screen resolution), and quality evaluation results in the form of raw logs.

[0206] According to one embodiment, the response log DB (808) may be partitioned or archived to improve large-scale data processing and analysis performance. Partitioning may refer to a method of dividing and storing a large-scale table based on a date, survey ID, or panel ID. Archiving may refer to a method of moving data that has been stored for a certain period (e.g., 1 year, 2 years) to a separate storage to maintain the query and analysis performance of the current database.

[0207] According to one embodiment, the system (400) can analyze behavioral patterns that appear during the process of a panel responding to a survey to infer the panel's attribute values ​​in real time and automatically detect whether there is a discrepancy with self-entered attribute values. Self-entered attribute values ​​may refer to core attribute values ​​including gender, age, residence, education level, income level, occupation group, and political orientation that the panel directly entered at the time of joining or in a profiling survey. The system (400) can verify the accuracy of attribute data stored in the panel database by extracting attribute information indirectly revealed in response behavior, without relying solely on self-entered attribute values.

[0208] According to one embodiment, the system (400) can analyze the response speed of a panel in a survey of a specific subject category to infer the panel's level of education or occupation. Response speed may refer to the time taken for the panel to read each question and select a response. The system (400) can query the response time per question recorded in the response log database and compare it with the average response time of all panelists who participated in the same survey. The system (400) can infer that a panelist who shows a significantly faster response speed than the average in questions requiring specialized knowledge is an expert familiar with the field or a person working in a related profession.

[0209] According to one embodiment, the system (400) can infer the level of education by analyzing the panel's understanding of the questions. Understanding of the questions may refer to the degree to which the panel accurately grasps the intent of the questions and selects appropriate responses. The system (400) can quantify the level of understanding of the questions by analyzing the consistency of responses to pairs of verification questions that ask the same content using different expressions. The system (400) can infer that a panel showing high consistency of responses to questions containing complex sentence structures or technical terms has a relatively high level of education.

[0210] According to one embodiment, the system (400) can analyze the vocabulary level, sentence complexity, and frequency of use of technical terms included in the free response text using natural language processing technology. Vocabulary level may refer to the difficulty and diversity of words used by the panel, and the system (400) can calculate a difficulty score for each word by referring to a vocabulary frequency database. Sentence complexity may refer to the length of the sentence, the number of subordinate clauses, and the complexity of the grammatical structure. The system (400) can calculate the estimated level of education of the panel by combining the vocabulary level score, the sentence complexity score, and the frequency of use of technical terms.

[0211] According to one embodiment, the system (400) can infer an income level by analyzing the response patterns shown by the panel in price-related questions. The price-related questions may include questions asking about the amount of money a panel is willing to pay for a specific product or service, the value assessment relative to the price, or changes in purchase intention upon a price increase. The system (400) can calculate a price sensitivity index by accumulating the response values ​​selected by the panel in price-related questions across multiple surveys. A panel with a low price sensitivity index and a consistently high purchase intention for high-priced products can be inferred to have a relatively high income level.

[0212] According to one embodiment, the system (400) can determine whether there is a discrepancy by comparing the inferred attribute value with the self-entered attribute value stored in the panel database. The system (400) can determine this as a potential discrepancy if the self-entered income level is recorded as a lower range even though the inferred income level corresponds to an upper range. The system (400) can determine this as a potential discrepancy if the difference between the inferred education level and the self-entered education level is greater than a predetermined grade.

[0213] According to one embodiment, the system (400) can calculate a discrepancy score that quantifies the degree of discrepancy between inferred attribute values ​​and self-written attribute values. If the discrepancy score exceeds a preset threshold, the system (400) can automatically classify the panel as a target for attribute re-verification. The system (400) can display a list of panels classified as targets for attribute re-verification on an administrator dashboard or automatically register them in an operator review queue.

[0214] According to one embodiment, the system (400) may send a re-verification survey containing additional verification questions regarding the attribute to a panel subject to attribute re-verification. The re-verification survey may include questions that re-verify the attribute in which a discrepancy was detected in various ways. For example, if an educational background discrepancy is detected, the system (400) may generate a re-verification survey including multiple questions asking about the highest level of education, major field, and year of graduation. The system (400) may make a final determination of the accuracy of the attribute value by combining the responses to the re-verification survey with the existing response history.

[0215] According to one embodiment, the system (400) can correct attribute values ​​in the panel database or adjust the panel reliability score based on the re-verification results. If the re-verification results confirm that the self-entered attribute value is accurate, the system (400) determines this as an error in the inference model and maintains the panel reliability score. If the re-verification results confirm that the self-entered attribute value is inaccurate, the system (400) updates the attribute values ​​in the panel database with the re-verification results and can lower the panel reliability score by a pre-set deduction factor.

[0216] According to one embodiment, the system (400) can build an attribute inference model by utilizing response behavior data accumulated over a plurality of surveys as training data for a machine learning model. The training data may include response times per question, selection patterns, text responses, and verified attribute values ​​of the corresponding panel stored in a response log database. The system (400) may perform preprocessing on the training data, including normalization, handling of missing values, and numerical transformation of categorical variables.

[0217] According to one embodiment, the attribute inference model may be implemented as a supervised learning model that takes a response behavior feature vector as input and outputs an estimated attribute value of a panel. The response behavior feature vector may be composed of a combination of numerical values ​​including average response time, variance of response time, linear response ratio, average length of text response, vocabulary diversity index, and price sensitivity index.

[0218] According to one embodiment, the system (400) may define the error between the prediction result of the attribute inference model and the actual verified attribute value as a loss function. The loss function may use cross-entropy loss for classification problems and mean squared error for regression problems. The system (400) may continuously improve inference accuracy by retraining the attribute inference model whenever new re-verification results are collected or periodically.

[0220] FIG. 9 is a block diagram showing the hierarchical structure of a panel management system according to one embodiment.

[0221] According to one embodiment, a panel management system (e.g., the system (400) of FIG. 4) may be configured with a multilayer structure including a data collection layer (910), a database layer (920), an artificial intelligence analysis layer (930), and an application service layer (940). Each layer may interact with the others to perform functions such as panel management, sampling, quality evaluation, and result prediction. The types and configurations of the layers are merely examples and are not limited thereto, and layers may be added or integrated depending on the system design.

[0222] According to one embodiment, the data collection layer (910) may refer to a layer that collects raw data necessary for the operation of a panel management system. The data collection layer (910) may collect membership sign-up and login events, basic profile input events, survey response events, community post creation events, and comment creation events.

[0223] According to one embodiment, the data collection layer (910) can receive event data transmitted from a web or mobile client in real time. The data collection layer (910) can add timestamps, session IDs, and device information to the received event data and transmit it to the database layer (920).

[0224] According to one embodiment, the database layer (920) may refer to a layer that structures and stores data collected from the data collection layer (910). The database layer (920) may include a matching DB, a member DB, a panel DB, and a response log DB. The detailed structure for each database of the database layer (920) is as described in FIG. 8.

[0225] According to one embodiment, the database layer (920) may be implemented using at least one of a relational database (e.g., MySQL, PostgreSQL) or a NoSQL database (e.g., MongoDB, Cassandra). The database layer (920) may provide transaction management, backup, and replication functions to ensure data integrity, consistency, and availability.

[0226] According to one embodiment, the artificial intelligence analysis layer (930) may refer to a layer that analyzes data stored in the database layer (920) and utilizes an artificial intelligence model to perform panel reliability evaluation, quality evaluation, sampling, and result prediction. The artificial intelligence analysis layer (930) may include a reliability evaluation module, a quality evaluation module, an automatic sampling module, and a digital twin and prediction module.

[0227] According to one embodiment, the reliability evaluation module can calculate a panel reliability score and detect attribute discrepancies based on the panel's attribute data, response history data, and behavioral data. The quality evaluation module can receive a survey response stream, evaluate response time, selection patterns, logical consistency, and text quality, and calculate a response quality score. The automatic sampling module can automatically extract a sample using survey request conditions, panel reliability, response history, and community activity history. The digital twin and prediction module can generate a digital twin profile that quantifies the panel's response propensity and predict results for a new survey based on this profile.

[0228] According to one embodiment, the artificial intelligence analysis layer (930) may utilize high-performance computing resources, including GPUs or TPUs, for training and inference of machine learning models. The artificial intelligence analysis layer (930) may operate a training pipeline for model training and a serving pipeline for real-time inference separately.

[0229] According to one embodiment, the application service layer (940) may refer to a layer that provides services to end users based on the analysis results of the artificial intelligence analysis layer (930). The application service layer (940) may include a survey operation service, a reward management service, and an administrator dashboard service.

[0230] According to one embodiment, the survey operation service may provide functions for survey creation, distribution, response collection, and result analysis. The survey operation service may provide an interface that allows the survey requester to design a survey and distribute it to a target panel. The reward management service may provide a function to automatically pay points or rewards to panelists upon completion of a survey response, and to exchange accumulated points for digital gift certificates. The administrator dashboard service may visualize panel distribution, sampling results, quality indicators, and warning notifications and provide them to the system administrator.

[0231] According to one embodiment, the application service layer (940) may interact with an external system via a RESTful API or a GraphQL API. The application service layer (940) may communicate with a web frontend (e.g., React, Vue.js) and a mobile application (e.g., iOS, Android) to provide a user interface.

[0232] According to one embodiment, the system (400) predicts the final quality score of the response before the survey response is completed, and if the predicted quality score is below a standard, it may preemptively invite a replacement sample before the response is completed. The final quality score may refer to a survey-unit response quality score calculated after the survey is completed by a quality evaluation module that combines response time, selection patterns, logical consistency, and text quality. Unlike the conventional method of retrospectively excluding responses from panels predicted to have low response quality after the survey is completed and recruiting additional samples, the system (400) can shorten the survey completion time by predicting quality during the response process.

[0233] According to one embodiment, the system (400) can predict a final response quality score based on real-time behavioral data collected while the panel responds to the initial questions of the survey. The initial questions may refer to questions corresponding to a pre-set proportion from the beginning of the total survey questions, for example, 20% of the total survey questions or the first 10 questions. The system (400) can collect behavioral data recorded in real-time in a response log database while the panel responds to the initial questions.

[0234] According to one embodiment, real-time behavioral data collected by the system (400) may include the average response time in the initial questions, whether a linear response pattern occurs, the number of page exits, and the frequency of skipping questions. The average response time may refer to the average value of the response time taken for each of the initial questions. Whether a linear response pattern occurs may refer to whether the same option is repeated consecutively more than a preset number of times in the matrix questions among the initial questions. The number of page exits may refer to the number of times the panel switched from the survey page to another web page or application. The frequency of skipping questions may refer to the proportion of questions that were skipped without being answered among the answerable questions.

[0235] According to one embodiment, the system (400) can convert collected real-time behavioral data into an initial behavioral feature vector. The initial behavioral feature vector may be composed of a combination of numerical values ​​including average response time, standard deviation of response time, number of items with linear responses, number of page abandonments, frequency of item skipping, and initial response completion rate. The system (400) can perform normalization on the initial behavioral feature vector to convert it into a form that can be input into a quality prediction model.

[0236] According to one embodiment, the system (400) can predict the final response quality score of the corresponding panel by inputting an initial behavioral feature vector into a quality prediction model. The quality prediction model may be implemented as a regression model that takes the initial behavioral feature vector as input and outputs a predicted quality score. Alternatively, the quality prediction model may be implemented as a binary classification model that outputs whether the final quality score is above a threshold.

[0237] According to one embodiment, a quality prediction model can be constructed by utilizing past survey response data as training data. The training data may consist of pairs of an initial behavioral feature vector extracted from past responses and the actual final quality score of the corresponding response. The system (400) can extract response records corresponding to the initial questions from past response data stored in a response log database and calculate an initial behavioral feature vector for each response. The system (400) can construct a training dataset by pairing the calculated initial behavioral feature vector with the final quality score of the corresponding response.

[0238] According to one embodiment, the system (400) can train a quality prediction model using a training dataset. The system (400) may use any one of linear regression, gradient boosting, or a neural network model as the quality prediction model. The system (400) defines the mean squared error between the predicted quality score and the actual quality score as the loss function and can adjust the weight parameters of the model in a direction that minimizes the loss function through gradient descent. The system (400) can perform cross-validation to evaluate the generalization performance of the quality prediction model and prevent overfitting.

[0239] According to one embodiment, the system (400) may determine that a response is highly likely to be ultimately excluded from the valid sample if the predicted quality score is below a preset threshold. The threshold score may refer to the minimum quality score for which the quality evaluation module recognizes a response as a valid sample and may be set by a system administrator. The system (400) may trigger an invitation for an alternative sample if the predicted quality score is lower than the threshold score plus a predetermined margin.

[0240] According to one embodiment, when an invitation to a replacement sample is triggered, the system (400) may select a replacement panel through a sampling module and send an invitation to participate in the survey. When selecting a replacement panel, the system (400) may prioritize selecting a panel that satisfies the same quota conditions as the original panel. Quota conditions may refer to a combination of demographic conditions, such as gender, age group, and region, included in the survey request conditions. When selecting a replacement panel, the system (400) may select from among panels whose panel reliability score is equal to or greater than the minimum reliability standard.

[0241] According to one embodiment, the system (400) can calculate an actual final quality score through a quality evaluation module after the original panel's response is completed. The system (400) can evaluate whether the prediction was accurate by comparing the actual quality score with the predicted quality score. If the actual quality score is above a standard, the system (400) can adopt the original panel's response as a valid sample. In this case, the system (400) can convert the response of a panel that completed the response later among the replacement panels into a preliminary sample, or exclude the response if the quota has already been met.

[0242] According to one embodiment, the system (400) may exclude the original panel's response from the valid sample and adopt the replacement panel's response as the valid sample if the actual quality score is below a threshold. The system (400) may send an invitation to participate in the survey to additional replacement panels if the replacement panel has not yet completed the response. The system (400) may repeat the replacement sample invitation in this manner until the number of valid samples reaches the required number of samples.

[0243] According to one embodiment, the system (400) can continuously improve prediction accuracy by reflecting the error between the predicted result and the actual result in the retraining of the quality prediction model. The system (400) can add the initial behavioral feature vector of the response and the actual final quality score to the training dataset whenever a new response is completed. The system (400) can reflect the latest response patterns by periodically retraining the quality prediction model when the training dataset increases beyond a predetermined size.

[0244] According to one embodiment, the system (400) may calculate and display on an administrator dashboard indicators including the number of replacement invitations sent, the replacement success rate, and the reduction rate of survey completion time to evaluate the effectiveness of the preemptive sample replacement function. The replacement success rate may refer to the ratio of cases where replacement was required because the original response was found to be below a standard among the number of replacement invitations sent. Through this preemptive sample replacement function, the system (400) can minimize the additional recruitment process due to a shortage of valid samples after the survey is completed, reduce the overall survey time, and improve the reliability of the survey results.

[0246] According to one embodiment, the system (400) can detect a problematic network by modeling the relationships between panels as a graph structure. A panel graph may refer to a graph data structure in which panels, surveys, community activities, and external authentication sources are nodes, and participation or sharing relationships between each node are edges.

[0247] According to one embodiment, the system (400) can connect multiple panel accounts sharing the same IP address, device fingerprint, or cookie to the edge in the panel graph. A device fingerprint may refer to a technology that uniquely identifies a specific device by combining information such as screen resolution, installed fonts, and browser plugins collected from a web browser or mobile application.

[0248] According to one embodiment, the system (400) can detect a cluster of nodes connected by an abnormally large number of edges in a panel graph. The system (400) can identify a group of accounts suspected of being operated by the same person creating multiple accounts by applying a graph analysis algorithm. The system (400) can perform additional verification on the panels included in the identified account group or exclude them from the sampling in bulk.

[0249] According to one embodiment, the system (400) can automatically pay points or rewards to the respondent upon completion of the survey response. The system (400) checks whether the survey has been completed in the response log database, and if the response is determined to be valid based on the quality evaluation results, it can credit pre-set points to the reward account of the panelist.

[0250] According to one embodiment, the system (400) may provide a reward management function that allows accumulated points to be exchanged for digital gift certificates or cash equivalents. The system (400) may help panelists view accumulated points through a mobile application or web interface and exchange them for desired gift certificates. The system (400) may increase the panelists' motivation to participate and induce sincere responses by providing rewards.

[0251] According to one embodiment, the system (400) may operate a plurality of databases including a matching database, a member database, a panel database, and a response log database. A matching database may refer to a database for mapping each user's nickname, internal unique number, and external authentication ID. An external authentication ID may include a social login ID or a mobile phone authentication token.

[0252] According to one embodiment, the matching database can serve as a key for consistently identifying the same user across a plurality of services, including panel management, community, and reward systems. The system (400) can integrate and analyze data generated from different services for the same user through the matching database.

[0253] According to one embodiment, the member database may store name, gender, date of birth, contact information, sign-up path, and consent information. The consent information may include consent for the collection and use of personal information, consent for provision to third parties, and consent for profiling. The system (400) may protect personal information by encrypting and storing personal identification information and using only internal IDs for panel analysis and survey analysis. The response log database may store survey IDs, question IDs, response content, response time, device information, and quality evaluation results in the form of raw logs.

[0255] Although the examples above have been illustrated with specific drawings, experts in the field may apply various technical modifications and variations based on them. For instance, appropriate results can be obtained even if the described techniques are performed differently from the presented order, or if the components of the described systems, structures, devices, or circuits are combined in different forms or replaced with other components or equivalents.

Claims

Claim 1 The panel management system includes memory and a processor for storing instructions, and when the instructions are executed by the processor, the system collects attribute data including gender, age, residence, education level, income level, occupation group, and political orientation through the panel candidate's registration information and profiling survey and stores it in a panel database; collects response history data and behavioral data including response content, response time, and selection patterns while the panel candidate participates in the survey; calculates a reliability score for each panel based on the attribute data, response history data, and behavioral data; calculates a panel reliability score by weighted summing the temporal consistency of the attribute data, the frequency of participation and response quality score history for each survey included in the response history data, and the average response speed, completion rate, and linear response rate included in the behavioral data; classifies each panel into one of a plurality of reliability grades based on the calculated panel reliability score; detects attribute inconsistencies by determining whether the change is a combination that is not realistically possible according to predefined rules when the value of the attribute data changes within a predetermined period; and sets target demographics and required sample size as survey request conditions. It receives minimum reliability criteria and automatically selects target panels as a sample using the reliability scores for each panel mentioned above; while collecting survey responses from the sample, it evaluates response time, selection patterns, logical consistency, and text quality to calculate a response quality score; records response times per item and per page; calculates the average response time per item for all panels participating in the same survey; detects a specific panel's response time as abnormal if it is below a preset lower limit ratio or above a preset upper limit ratio compared to the average response time; and determines whether a response is insincere by analyzing linear response patterns in which the same option number is repeated consecutively more than a preset number of times in matrix items.A system that checks logical consistency by determining whether the response values ​​for the first question and the second question within the same survey correspond to a predefined mutually exclusive condition rule, calculates a text quality score by analyzing whether the number of characters entered for the free response text is greater than or equal to a pre-set minimum number of characters, whether the semantic similarity between the question sentence and the response sentence is greater than or equal to a pre-set threshold, and whether meaningless strings or repetitive strings are included, calculates a survey-unit response quality score by synthesizing the abnormal response time detection result, the linear response pattern determination result, the logical consistency check result, and the text quality score, and controls the update of the reliability score per panel based on the response quality score. Claim 2 In claim 1, the above instructions are a system that controls the system to lower the reliability rating of the panel or switch it to a blacklist state when the above response quality score accumulates below a standard for a predetermined period of time when executed by the processor. Claim 3 In claim 1, the above instructions, when executed by the processor, are a system that, when the system detects the above attribute mismatch, compares the attribute value responded to in a first survey by a panel identified by the same panel identifier with the attribute value responded to in a second survey performed within a predetermined period, determines whether the change in the attribute value resulting from the comparison corresponds to a predefined impossible change rule as a type of change that cannot occur in a normal life course within the predetermined period, and if it is determined to correspond to the impossible change rule, excludes the survey response from the analysis target and lowers the reliability score of the panel by a pre-set deduction coefficient, and if the attribute mismatch corresponding to the impossible change rule accumulates more than a predetermined number of times, controls the system to switch the panel to an unreliable state and automatically exclude it from the subsequent sampling process. Claim 4 In claim 1, the above instructions, when executed by the processor, control the system to automatically select the above-mentioned survey target panel as a sample by receiving a target condition as a survey request condition that includes at least one of the following: possession of a specific product, experience using a specific service, performance of a specific activity, or possession of a specific interest; searching past survey response history stored in the panel database to extract a first candidate group of panels that have consistently provided responses matching the above-mentioned target condition multiple times; searching a community activity history database to extract a second candidate group of panels that have a history of writing posts on topics related to the above-mentioned target condition or writing comments on posts on such topics; integrating the first candidate group and the second candidate group to create a survey target candidate group, thereby securing a set of panels pre-verified as matching the above-mentioned target condition without distributing separate screening questions; and checking the reliability score of each panel included in the survey target candidate group to select a final sample from among panels that are above the minimum reliability standard. Claim 5 delete Claim 6 In claim 1, when the instructions are executed by the processor, the system generates a digital twin profile in the form of a multidimensional vector that quantifies the response propensity of the panel by taking attribute data, topic-specific response history data, response quality score history, and behavioral data of each panel as input; the digital twin profile includes response propensity parameters including response distribution by topic category, preference choice tendencies, positive or negative response ratios, and price sensitivity indices of surveys in which the panel participated in the past; when a new survey is planned, the system generates a survey characteristic vector by analyzing the item structure, topic category, choice type, and response scale of the new survey; calculates the similarity between the survey characteristic vector and a plurality of digital twin profiles to simulate what response distribution each digital twin profile will exhibit for the new survey; improves prediction accuracy by periodically updating the weight parameters of the prediction model using the error between the actual response results of similar surveys conducted in the past and the prediction results based on the digital twin profile at that point in time as training data; and, based on the simulated response distribution, the expected preference distribution, expected purchase intention ratio, or expected conversion rate for the new survey A system that controls the generation of a survey result prediction report including at least one of the following. Claim 7 In a method for online survey sampling, quality assessment, and result prediction using a panel management system, the operation of collecting attribute data including gender, age, residence, education level, income level, occupation group, and political orientation through the registration information and profiling survey of panel candidates and storing it in a panel database; the operation of collecting response history data and behavioral data including response content, response time, and selection pattern while the panel candidate participates in the survey; the operation of calculating a reliability score for each panel based on the attribute data, response history data, and behavioral data; the operation of calculating the panel reliability score by weighting the sum of the temporal consistency of the attribute data, the frequency of participation and response quality score history for each survey included in the response history data, and the average response speed, completion rate, and linear response rate included in the behavioral data; the operation of classifying each panel into one of a plurality of reliability grades based on the calculated panel reliability score; the operation of detecting attribute inconsistency by determining whether the change is a combination that is not realistically possible according to a predefined rule when the value of the attribute data changes within a predetermined period; and as survey request conditions, target demographics, required sample size, and An operation of receiving a minimum reliability criterion as input and automatically selecting a survey target panel as a sample using the reliability score for each panel, an operation of calculating a response quality score by evaluating response time, selection patterns, logical consistency, and text quality while collecting survey responses from the sample, an operation of recording response times per item and per page, calculating the average response time per item for all panels participating in the same survey, and detecting as an abnormal response time if the response time of a specific panel is below a preset lower limit ratio or above a preset upper limit ratio compared to the average response time.A method comprising: an operation to determine whether a response is insincere by analyzing a linear response pattern in which the same option number is repeated consecutively more than a preset number of times in a matrix item; an operation to check logical consistency by determining whether the response value for a first item and the response value for a second item within the same survey correspond to a preset mutually exclusive condition rule; an operation to calculate a text quality score by analyzing whether the number of characters entered for a free response text is greater than or equal to a preset minimum number of characters, whether the semantic similarity between the question sentence and the response sentence is greater than or equal to a preset threshold, and whether meaningless strings or repetitive strings are included; an operation to calculate a survey-unit response quality score by synthesizing the abnormal response time detection result, the linear response pattern determination result, the logical consistency check result, and the text quality score; and an operation to update the reliability score for each panel based on the response quality score. Claim 8 A method according to claim 7, further comprising a control operation to lower the reliability rating of the panel or switch it to a blacklist state if the response quality score accumulates below a standard for a predetermined period.

Images