Questionnaire quality assessment method and system thereof

Abstract

A survey quality evaluation method according to an embodiment of the present disclosure may include the steps of: estimating a response time for a survey including one or more items, calculating a score for each of a plurality of evaluation items that are pre-set for the survey, calculating a quality score for the survey based on the response time and the score for each of the plurality of evaluation items, and outputting feedback on the survey to a user based on the calculated quality score of the survey.

Description

Technology Field

[0001] The present disclosure relates to a method and system for evaluating survey quality, and more specifically, to a method for automating the evaluation of survey quality based on the composition of items included in the survey. Background Technology

[0002] Recently, online surveys for data collection and analysis are being actively conducted across various fields. While online surveys are efficient as they allow for the collection of large-scale responses without time or location constraints, they often fail to provide clear feedback on the ability to quickly evaluate survey design quality and secure high-quality data. The lack of quality evaluation results and specific issue analyses makes it difficult to determine whether to proceed with a survey or accept proposed improvements. Furthermore, the insufficient functionality to easily modify surveys to incorporate suggested improvements causes users to struggle with efficiently optimizing their surveys. Prior art literature

[0003] Published Patent Application No. 10-2013-0055726 (Date of publication: May 29, 2013) The problem to be solved

[0004] The technical problem to be solved by the embodiments of the present disclosure is to provide an automated method for evaluating the quality of a survey created by a user to determine whether it is well-designed enough to obtain high-quality data.

[0005] In addition, the technical problem to be solved through the embodiments of the present disclosure is to provide a method for immediately presenting to the user improvement measures that can improve the quality of the survey based on the results of the survey quality evaluation. means of solving the problem

[0006] A survey quality evaluation method according to an embodiment of the present disclosure may include the steps of: estimating a response time for a survey including one or more items, calculating a score for each of a plurality of evaluation items that are pre-set for the survey, calculating a quality score for the survey based on the response time and the score for each of the plurality of evaluation items, and outputting feedback on the survey to a user based on the calculated quality score of the survey.

[0007] In one embodiment, the step of estimating the response time for the survey may include the step of estimating the response time for each of the one or more items, and the step of summing the estimated response times for each of the one or more items.

[0008] In one embodiment, the step of estimating the response time for each of the one or more items comprises: determining whether the explanatory power of a first model for a first item among the one or more items is greater than or equal to a preset threshold, wherein the first model is a model trained to estimate the response time by receiving input of an item having the same type as the first item; if the explanatory power is greater than or equal to the preset threshold, inputting the first item into the first model to estimate the response time; and if the explanatory power is less than the preset threshold, estimating the response time based on the type of the first item, wherein the step of estimating the response time based on the type of the first item comprises: determining whether there exists a preset number or more of first response time data pre-estimated for items having the same type as the first item; if the first response time data exists greater than or equal to the preset number, estimating the average of the first response time data as the response time of the first item; and if the first response time data does not exist greater than or equal to the preset number, the first It may include a step of estimating the average of the second response time data, which is previously estimated for a question having a second type set to correspond to the type of question, as the response time of the first question.

[0009] In one embodiment, the step of estimating the response time based on the type of the first question may further include the step of estimating the median of the stored total response time data as the response time of the first question when the first response time data does not exist in a number greater than the preset number and there is no second type preset to correspond to the type of the first question.

[0010] In one embodiment, the step of calculating the score for each of the plurality of evaluation items may include the step of calculating the score for each of the first plurality of evaluation items pre-set for each of the one or more questions, and the step of calculating the score for each of the second plurality of evaluation items pre-set for the entire survey.

[0011] In one embodiment, the first plurality of evaluation items include the length of each of the one or more questions, the number of options, the number of characters in the options, and the number of options including images, and the second plurality of evaluation items may include the number of subjective questions.

[0012] In one embodiment, the first plurality of evaluation items further include the appropriateness of the language used in each of the one or more items, and the step of calculating the score for each of the first plurality of evaluation items may include the step of inputting a prompt requesting to score the appropriateness of the language of the survey into a language model to calculate the score for the appropriateness of the language.

[0013] In one embodiment, the second plurality of evaluation items further include whether the survey can be completed, and the step of calculating the score for each of the second plurality of evaluation items may include the step of calculating the score for whether the survey can be completed based on whether there exists a question that causes an infinite loop between the one or more questions.

[0014] In one embodiment, the step of calculating a score for each of the plurality of evaluation items may include receiving a selection for a first evaluation item among the plurality of evaluation items, and calculating a score for each of the remaining evaluation items excluding the first evaluation item among the plurality of evaluation items.

[0015] In one embodiment, the step of calculating the quality score of the survey may include the step of standardizing and summing the scores of the response time and each of the plurality of evaluation items, and the step of calculating the quality score of the survey by subtracting the summed score from the reference score.

[0016] A survey quality evaluation system according to another embodiment of the present disclosure includes a processor and a memory for storing instructions, and when the instructions are executed by the processor, the processor may perform the following operations: estimating a response time for a survey including one or more items; calculating a score for each of a plurality of evaluation items that are pre-set for the survey; calculating a quality score for the survey based on the response time and the score for each of the plurality of evaluation items; and outputting feedback for the survey to a user based on the calculated quality score for the survey.

[0017] In one embodiment, the operation of estimating the response time for the survey may include the operation of estimating the response time for each of the one or more items, and the operation of summing the estimated response times for each of the one or more items.

[0018] In one embodiment, the operation of estimating the response time for each of the one or more items comprises: determining whether the explanatory power of a first model for a first item among the one or more items is greater than or equal to a preset threshold, wherein the first model is a model trained to estimate the response time by receiving input of an item having the same type as the first item; if the explanatory power is greater than or equal to the preset threshold, inputting the first item into the first model to estimate the response time; and if the explanatory power is less than the preset threshold, estimating the response time based on the type of the first item, wherein the operation of estimating the response time based on the type of the first item comprises: determining whether there exists a preset number or more of first response time data pre-estimated for items having the same type as the first item; if the first response time data exists greater than or equal to the preset number, estimating the average of the first response time data as the response time of the first item; and if the first response time data does not exist greater than or equal to the preset number, the It may include an operation of estimating the average of the second response time data, which is previously estimated for a question having a second type set to correspond to the type of question 1, as the response time of the first question.

[0019] In one embodiment, the operation of estimating the response time based on the type of the first question may further include the operation of estimating the median of the stored total response time data as the response time of the first question when the first response time data does not exist in a number greater than the preset number and there is no second type preset to correspond to the type of the first question.

[0020] In one embodiment, the operation of calculating the score for each of the plurality of evaluation items may include the operation of calculating the score for each of the first plurality of evaluation items pre-set for each of the one or more items, and the operation of calculating the score for each of the second plurality of evaluation items pre-set for the entire survey.

[0021] In one embodiment, the first plurality of evaluation items include the length of each of the one or more questions, the number of options, the number of characters in the options, and the number of options including images, and the second plurality of evaluation items may include the number of subjective questions.

[0022] In one embodiment, the first plurality of evaluation items further include the appropriateness of the language used in each of the one or more items, and the operation of calculating the score for each of the first plurality of evaluation items may include inputting a prompt requesting to score the appropriateness of the language of the survey into a language model to calculate a score for the appropriateness of the language.

[0023] In one embodiment, the second plurality of evaluation items further include whether the survey can be completed, and the operation of calculating the score for each of the second plurality of evaluation items may include the operation of calculating the score for whether the survey can be completed based on whether there is a question that causes an infinite loop between the one or more questions.

[0024] In one embodiment, the operation of calculating a score for each of the plurality of evaluation items may include receiving a selection for a first evaluation item among the plurality of evaluation items, and calculating a score for each of the remaining evaluation items excluding the first evaluation item among the plurality of evaluation items.

[0025] In one embodiment, the operation of calculating the quality score of the survey may include the operation of standardizing and summing the scores of the response time and each of the plurality of evaluation items, and the operation of calculating the quality score of the survey by subtracting the summed score from the reference score.

[0026] A computer program according to another embodiment of the present disclosure may be combined with a computing device and stored in a computer-readable recording medium to execute the steps of: estimating a response time for a survey including one or more items; calculating a score for each of a plurality of evaluation items set for the survey; calculating a quality score for the survey based on the response time and the score for each of the plurality of evaluation items; and outputting feedback on the survey to a user based on the calculated quality score of the survey. Brief explanation of the drawing

[0027] FIG. 1 is a block diagram showing the configuration of a survey service providing system according to an embodiment of the present disclosure. FIG. 2 exemplarily shows a survey editing screen according to an embodiment of the present disclosure. FIG. 3 exemplarily shows a survey quality evaluation screen according to an embodiment of the present disclosure. Figure 4 shows a case where feedback on a specific item is reviewed in detail on the survey quality evaluation screen of Figure 3. FIG. 5 illustrates an exemplary survey editing screen according to another embodiment of the present disclosure. FIG. 6 exemplarily illustrates the output of a predicted result when a survey is modified according to the survey quality evaluation results according to an embodiment of the present disclosure. FIG. 7 is a flowchart exemplarily illustrating a survey quality evaluation method according to an embodiment of the present disclosure. Figure 8 is a flowchart specifically illustrating the steps for estimating the response time to the survey in Figure 7. Figure 9 is a flowchart specifically illustrating the steps for estimating the response time for each of one or more items of Figure 8. Figure 10 is a flowchart specifically illustrating the steps for estimating response time based on the type of the first question in Figure 9. FIG. 11 is a flowchart illustrating an example of the step of calculating the score for each of the multiple evaluation items of FIG. 7. FIG. 12 is a flowchart illustrating another embodiment of the step of calculating the score for each of the multiple evaluation items of FIG. 7. Figure 13 is a flowchart specifically illustrating the steps for calculating the quality score of the survey in Figure 7. FIG. 14 is a block diagram showing the hardware configuration of a computing device for survey quality evaluation according to an embodiment of the present disclosure. Specific details for implementing the invention

[0028] Preferred embodiments of the present disclosure are described in detail below with reference to the accompanying drawings. The advantages and features of the present disclosure and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present disclosure is not limited to the embodiments described below but may be implemented in various different forms. The embodiments are provided merely to make the present disclosure complete and to fully inform those skilled in the art of the scope of the invention, and the present disclosure is defined only by the scope of the claims.

[0029] It should be noted that when assigning reference numerals to the components of each drawing, the same components are given the same reference numeral whenever possible, even if they are shown in different drawings. Furthermore, in describing the present disclosure, if it is determined that a detailed description of related known components or functions could obscure the essence of the present disclosure, such detailed description is omitted.

[0030] Unless otherwise defined, all terms used herein (including technical and scientific terms) may be used in a meaning commonly understood by those skilled in the art to which this disclosure pertains. Furthermore, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless explicitly and specifically defined otherwise. The terms used herein are for describing embodiments and are not intended to limit the invention. In this specification, the singular form includes the plural form unless specifically stated otherwise in the text.

[0031] Additionally, terms such as first, second, A, B, (a), (b), etc., may be used to describe the components of the present disclosure. These terms are intended only to distinguish the components from other components, and the nature, order, or sequence of the components is not limited by the terms. Where it is stated that a component is "connected," "coupled," or "joined" to another component, it should be understood that the component may be directly connected or joined to the other component, but that another component may also be "connected," "coupled," or "joined" between each component.

[0032] As used in the specification, "comprises" and / or "comprising" do not exclude the presence or addition of one or more other components, steps, actions, and / or elements to the mentioned components, steps, actions, and / or elements.

[0033] FIG. 1 is a block diagram showing the configuration of a survey service providing system (10) according to an embodiment of the present disclosure. Referring to FIG. 1, the survey service providing system (10) may include a user terminal (100) and a service server (200).

[0034] A user terminal (100) may be a computing device used by a user to receive a survey service from a service server (200). In this specification, a survey service refers to a service for a user to create a survey containing one or more questions and to collect responses from other users. A survey service according to an embodiment of the present disclosure may include a survey quality evaluation service for evaluating the quality of the created survey. Furthermore, unless otherwise stated, in this specification, a user is assumed to mean a survey creator and other users are assumed to mean survey respondents.

[0035] As illustrated exemplarily in FIG. 1, the user terminal (100) may include a smartphone (100a), a desktop (100b), a laptop (100c), etc. However, the present disclosure is not limited thereto, and the user terminal (100) may be implemented as any device. In the following, any user terminal (e.g., 100a) or all user terminals (e.g., 100a to 100c) collectively will be denoted by reference numeral 100.

[0036] The service server (200) may refer to a computing device / system that provides survey services. For example, the service server (200) may refer to a device / system that provides an online survey creation site to a user terminal (100). The service server (200) can complete a survey by receiving input from the user regarding questions, question types, options, etc., and can manage and evaluate the quality of the completed survey.

[0037] Meanwhile, in the present disclosure, the service server (200) providing a survey service to a user may mean rendering a page / screen for survey creation and quality evaluation on the screen of the user terminal (100) (e.g., the screen of a web browser or application running on the user terminal (100)). Accordingly, in the following specification, rendering a specific component will be understood as providing or displaying the corresponding specific component on the screen of the user terminal (100).

[0038] The service server (200) may provide the user with an interface for editing and creating surveys. Through the provided interface, the user can determine the type for each item of the survey, input questions and options, and add images as needed. For example, the types of items can be broadly classified into multiple-choice and open-ended types; multiple-choice items may include single-choice, multiple-choice, and ranking types, while open-ended items may include short-answer, descriptive, and number / address / phone number / date / email / name / date of birth types. However, the present disclosure is not limited thereto, and the types of items may be classified in a more diverse manner. An interface for editing and creating surveys is described later with reference to FIG. 2.

[0039] The service server (200) can evaluate the quality of a survey created through such an interface in response to a user's request. In the present disclosure, evaluating the quality of a survey may include all processes of estimating the response time of the survey, analyzing the number of questions, the number of options per question, and the number of characters per option, determining whether the created survey contains elements that cause fatigue to other users who will respond to the survey, and suggesting modifications to reduce fatigue to the user (survey creator) as feedback. Below, we will review embodiments related to the evaluation of survey quality.

[0040] A service server (200) can estimate the response time for a survey that includes one or more questions. First, the service server (200) can estimate the response time for each of the one or more questions, and then sum the estimated response times for each question to estimate the response time for the entire survey. Hereinafter, one of the one or more questions included in the survey will be referred to as the first question, and it will be understood that the method for estimating the response time for the first question is applied equally to estimating the response time for all remaining questions.

[0041] For the first question, the service server (200) can determine whether the explanatory power of a regression analysis model trained to estimate response time by receiving a question of the same type as the first question is greater than or equal to a preset threshold (e.g., 0.2 or greater). The regression analysis model may be a model trained to estimate the response time of the first question through a weighted sum of scales related to the question and the answer choices for each type of question, and the explanatory power of the regression analysis model may refer to the R-squared value of the regression analysis model. For example, if the first question is a single-choice multiple-choice question, the regression analysis model may be a model trained to estimate the response time of the first question through the number of answer choices, whether there are answer choices including images, the number of characters in the question, and the weighted sum of the number of characters in the answer choices. The regression analysis model may be updated at a preset cycle (e.g., 6 months or 1 year) for each type of question, and the explanatory power may be determined along with the update. If it is determined that the explanatory power is greater than or equal to a preset threshold, the service server (200) can input the first question into a corresponding regression analysis model to estimate the response time.

[0042] On the other hand, if the explanatory power of the regression analysis model corresponding to the type of the first question is below a preset threshold, the service server (200) can estimate the response time based on the type of the first question. First, the service server (200) can determine whether there is a preset number (e.g., 30) or more of the first response time data that has been estimated for other questions having the same type as the first question. That is, the first response time data may correspond to data in which the estimated response time for questions having the same type as the first question in a survey other than the survey containing the first question is accumulated.

[0043] If there are more than a preset number of first response time data, the service server (200) can estimate the average of the first response time data as the response time of the first question. For example, if the first question is a short-answer question and the explanatory power of the regression analysis model corresponding to the short-answer question is below a preset threshold, but there are more than a preset number of pre-estimated response time data for the short-answer question, the response time of the first question can be estimated as the average of the pre-estimated response time data for the short-answer question. However, if there are not more than a preset number of first response time data, it may be difficult to estimate the response time of the first question as the average of the response times estimated for questions of the same type as the first question.

[0044] In such a case, the service server (200) can estimate the response time of the first question by estimating the average of the second response time data, which is pre-set for a question having a second type that corresponds to the type of the first question. For example, if the type of the first question is a subjective short-answer ranking type, the subjective short-answer ranking type can be pre-set to correspond to the subjective short-answer question. Since the subjective short-answer ranking type can correspond to cases where there are multiple options in the subjective short-answer question, the response time of the first question can be estimated by multiplying the response time data pre-estimated for the subjective short-answer question by the number of options.

[0045] Finally, if the explanatory power of the regression learning model corresponding to the type of the first question is below a preset threshold, and there are no preset numbers of first response time data preset for questions of the same type as the first question, and there are no preset numbers of second types preset to correspond to the first question, the service server (200) can estimate the median of the preset total response time data as the response time of the first question.

[0046] As described above, the response time for the first item can be estimated by using a regression analysis model corresponding to the type of the first item, using the average of previously estimated response times for items having the same type as the first item, using the average of previously estimated response times for items having a different type set to correspond to the type of the first item, or using the median of the total response times. Then, after the response time is estimated for each of one or more items using the same method, the result of summing the estimated response times can be estimated as the response time for the entire survey.

[0047] Afterward, the service server (200) can calculate the score for each of the multiple evaluation items that are pre-configured for the survey. The evaluation items for the survey can be classified into evaluation items at the item level and evaluation items at the entire survey level. Evaluation items at the item level may include the number of characters in the item, the number of options, the number of characters in the options, the number of options containing images, the maximum number of characters in the options, the number of multiple responses for open-ended questions, etc., and evaluation items at the entire survey level may include the number of open-ended questions, the number of multiple evaluation questions, etc. Additionally, the previously estimated response time for the entire survey may also be considered as one of the evaluation items at the entire survey level.

[0048] The service server (200) can calculate scores for each of the evaluation items in the pre-set question unit for each of one or more questions, and can calculate scores for each of the evaluation items in the pre-set survey unit for the entire survey. Each score of an evaluation item can correspond to the value of the evaluation item itself. For example, if the number of options for the first question is 5, the score of the number of options evaluation item for the first question can be calculated as 5 points, and if the number of open-ended questions for the entire survey is 10, the score of the number of open-ended questions evaluation item for the entire survey can be calculated as 10 points. This is a score before standardization, and the process of standardizing the score will be described later when explaining the process of finally calculating the quality score of the survey.

[0049] In some embodiments, the item-unit evaluation criteria may further include the appropriateness of the language used in each of one or more items. Here, the appropriateness of the language may broadly include whether double negations are used, whether words of a difficulty level suitable for the subject of the survey are used, and whether a writing style suitable for the subject of the survey is used. The service server (200) may input a prompt requesting the appropriateness of the language of the survey into the language model to calculate a score for the appropriateness of the language.

[0050] Here, a language model refers to a neural network model that has acquired a universal understanding of language (or natural language / text) by learning a vast amount of text (e.g., texts from various domains); depending on the amount of data learned and the size of the parameters, the language model may also be referred to as a 'largescale language model (LLM).'

[0051] In some other embodiments, the evaluation item for the entire survey unit may further include whether the survey can be completed. For example, cases where the survey cannot be completed may include cases where the design of conditional items is incorrect, or cases where there are items that cause an infinite loop between one or more items included in the survey. For example, cases where the design of conditional items is incorrect may indicate a case where, if a specific option is selected in the first item, the user is instructed to respond to the second item, but the second item does not exist. And cases where there are items that cause an infinite loop may indicate a case where, if a specific option is selected in the first item, the user is instructed to respond to the second item, and if a specific option is selected in the second item, the user is instructed to respond to the first item. The service server (200) may calculate a score regarding whether the survey can be completed based on whether there are items that prevent the survey from being completed in this way.

[0052] In another embodiment, the service server (200) may receive a first evaluation item to be excluded from the evaluation from the user. This is to prevent the survey quality score from being calculated low because the options contain many images or the number of options is large, for example, in cases where images must be included in the options due to the nature of the survey, or where the options inevitably amount to dozens. The service server (200) may calculate the score for each of the remaining evaluation items excluding the first evaluation item.

[0053] The service server (200) can calculate a quality score of the survey based on the estimated response time and the scores calculated for each of the multiple evaluation items. To calculate the quality score of the survey, the service server (200) can standardize the scores of the response time and each of the multiple evaluation items. Here, standardization of scores means classifying the response time and the scores of each evaluation item into three grades according to pre-set criteria, and then converting them into scores corresponding to those grades. For example, Table 1 below illustrates the criteria for classifying the response time and the scores of some evaluation items into three grades (minor grade, major grade, critical grade).

[0054] Evaluation items No problem Minor majors Critical Character count of the question x<66 66≤x<121 121≤x<177 177≤x Number of options x<10 10≤x<19 19≤x<28 28≤x Number of options including images x<7 7≤x<13 13≤x<19 19≤x Maximum number of characters for the option x<17 17≤x<34 34≤x<52 52≤x Number of multiple responses to open-ended questions x<5 5≤x<10 10≤x<14 14≤x Number of subjective questions x<4 4≤x<7 7≤x<11 11≤x Response time (seconds) x<210 210≤x<390 390≤x<630 630≤x

[0055] For example, if the number of characters in a question is 66 or more but less than 121, the question may be determined to be a minor grade with a minor problem regarding the number of characters; if the number of open-ended questions is 7 or more but less than 11, the entire survey may be determined to be a major grade with a moderate problem regarding the number of open-ended questions; and if the estimated response time is 630 seconds or more, the entire survey may be determined to be a critical grade with a serious problem regarding the response time. Table 1 is for illustrative purposes only, and it goes without saying that the types of grades and the criteria for each grade are subject to change. Once an evaluation item is determined to be a minor grade, a major grade, or a critical grade, each grade can be converted into a corresponding score. For example, at the item level, a minor grade can be converted to 17 points, a major grade to 29 points, and a critical grade to 35 points, and at the entire survey level, a minor grade can be converted to 25 points, a major grade to 38 points, and a critical grade to 76 points. If there are no issues, the score for the item can be converted to 0 points at both the item level and the entire survey level. The scores converted in this way can be defined based on the range where the response time increases statistically significantly for each evaluation item.

[0056] Here, the interval where the response time increases statistically significantly can be determined by using the percentile of the value of each evaluation item as the interval where the response time increases statistically significantly among the interval where the value of the evaluation item is less than or equal to the first quartile (Q1), the interval where it is greater than the first quartile (Q1) and less than or equal to the second quartile (Q2), the interval where it is greater than the second quartile (Q2) and less than or equal to the third quartile (Q3), and the interval where it is greater than the third quartile (Q3).

[0057] The service server (200) can sum up all standardized scores in this manner. Specifically, the summation of standardized scores means the sum of the standardized scores for each of one or more items, the standardized scores for each item at the item level, the standardized scores for each evaluation item at the entire survey level, and the standardized scores for the estimated response time.

[0058] The service server (200) can calculate the quality score of the survey by subtracting the summed score from the reference score. More specifically, the service server (200) can classify the quality score calculated in this way into multiple grades according to a pre-set standard. For example, assuming the reference score is 100 points, if one major grade problem occurs at the item level and one minor grade problem occurs at the entire survey level, the quality score of the survey will be calculated as 100-29-25=46 points. For example, according to an embodiment of the present disclosure, if the quality score calculated in this way is 50 points or more, the survey can be determined as a good grade; if it is 25 points or more but less than 50 points, the survey can be determined as a fair grade; and if it is less than 25 points, the survey can be determined as a poor grade.

[0059] The service server (200) can output feedback on the survey to the user based on the quality score of the survey calculated in this way. The feedback on the survey may include the grade of the survey based on the calculated quality score, the items that have problems, the items that need to be corrected in the items that have problems, the items that need to be corrected in the entire survey, and the estimated response time when the survey is modified according to the provided feedback. Such feedback may be provided through a separate interface or may be provided in the form of a real-time popup in the survey editing interface.

[0060] The service server (200) may be implemented with at least one computing device. For example, all functions of the service server (200) may be implemented in a single computing device, or the first function of the service server (200) may be implemented in a first computing device and the second function may be implemented in a second computing device. Alternatively, specific functions of the service server (200) may be implemented in multiple computing devices. For example, the computing device may include all types of devices equipped with computation means and communication means.

[0061] The user terminal (100) can also perform various operations / functions to provide the aforementioned survey service. For example, the survey creation and quality evaluation service of the service server (200) can be provided through a web browser or application of the user terminal (100). For instance, the user terminal (100) can display the quality evaluation results of the created survey on the screen of the web browser or application in conjunction with the service server (200).

[0062] As illustrated in FIG. 1, the user terminal (100) and the service server (200) can communicate through a network. For example, the network can be implemented as any type of wired or wireless network, such as a Local Area Network (LAN), a Wide Area Network (WAN), a mobile radio communication network, or Wibro (Wireless Broadband Internet).

[0063] Up to now, with reference to FIG. 1, a survey service provided by a user terminal (100) and a service server (200) according to some embodiments of the present disclosure has been described in a schematic manner. Hereinafter, a survey quality evaluation method according to some embodiments of the present disclosure will be described through a specific user interface displayed on the user terminal (100). For example, the user interface may be a screen displayed on an application running on the user terminal (100) or on a web browser.

[0064] FIG. 2 illustrates an exemplary survey editing screen (20) according to an embodiment of the present disclosure. Referring to FIG. 2, a plurality of questions (21, 22) are displayed on the survey editing screen (20). Each question (21, 22) may consist of a question (21a, 22a) and an answer choice (21b, 22b). Through the survey editing screen (20) shown in FIG. 2, the user can add a question and an answer choice to add a single question. Once a survey is completed by adding all questions, the user can select the survey quality evaluation button (23), and the quality of the survey completed by the user can be evaluated according to the embodiments described with reference to FIG. 1. The results of the survey quality evaluation may be displayed on the survey quality evaluation screen, which will be described later with reference to FIG. 3.

[0065] FIG. 3 illustrates an exemplary survey quality evaluation screen (30) according to an embodiment of the present disclosure. Referring to FIG. 3, survey quality (31) and feedback (32, 33) are shown. Additionally, the number of questions and the estimated response time may be displayed to the right of the survey quality (31). Feedback (32) indicates that the question length is long (i.e., the number of characters in the question is long) and suggests reducing the number of characters in the question to 65 characters or less. Furthermore, feedback (33) indicates that the number of selectable options for a ranking question is large and suggests reducing the number of selectable options to 3 or less. The user can review the feedback in more detail by selecting the expansion buttons (32a, 33a) included in the feedback (32, 33), which will be described later with reference to FIG. 4. Additionally, the user can check the expected survey quality and response time when all items requiring review have been resolved by selecting the toggle button (34), which will be described later with reference to FIG. 6.

[0066] FIG. 4 illustrates a case where feedback on a specific item is reviewed in detail on the survey quality evaluation screen (30) of FIG. 3. The survey quality evaluation screen (30) of FIG. 4 may correspond to the case where the user selects the expansion button (32a) on the survey quality evaluation screen (30) of FIG. 3. Referring to FIG. 4, as a result of selecting the expansion button (32a), detailed review items (32b) regarding the feedback (32) are displayed. Through this, the user will be able to modify the item where the problem occurred more specifically.

[0067] FIG. 5 illustrates an exemplary survey editing screen (20) according to another embodiment of the present disclosure. Referring to FIG. 5, the survey editing screen (20) is identical to FIG. 2, but when the user selects the survey quality evaluation button (23), instead of moving to a separate survey quality evaluation screen (30) as in FIG. 3 and FIG. 4, a survey quality evaluation popup (24) is displayed. The survey quality evaluation popup (24) is a component that displays feedback on the currently selected question (e.g., question (21) in FIG. 5), and when the user selects the popup (24), the user may move to the survey quality evaluation screen (30) shown in FIG. 3 and FIG. 4.

[0068] Alternatively, depending on the case, even if the user does not select the survey quality evaluation button (23), the response time for the corresponding question may be estimated in real time and the score for the evaluation item may be calculated during the process of the user entering the question and option, and a survey quality evaluation popup (24) may be displayed based on that.

[0069] FIG. 6 exemplarily illustrates the output of predicted results when a survey is modified according to the survey quality evaluation results according to an embodiment of the present disclosure. Referring to FIG. 6, it is shown that the initial grade of the survey quality is poor, the number of items is 10, and the estimated response time is 4 minutes. When the user selects the toggle button (41), the survey quality, number of items, and estimated response time can be newly displayed when all items requiring review are resolved. In FIG. 6, it is shown that when items requiring review are resolved, the grade of the survey quality changes to good, the number of items changes from 10 to 8, and the estimated response time changes from 4 minutes to 2 minutes.

[0070] FIG. 7 is a flowchart exemplarily illustrating a survey quality evaluation method according to an embodiment of the present disclosure. For reference, FIGS. 7 and FIGS. 8 to 13, which will be described later, illustrate steps / operations of a product combination recommendation method performed on a user terminal (100). Therefore, in the following descriptions, if the subject of a specific step / operation is omitted, it can be understood as being performed on the user terminal (100). However, in an actual environment, some functions / operations of the user terminal (100) may be implemented on the service server (200) side, and some functions / operations of the service server (200) may be implemented on the user terminal (100) side. That is, the subject performing a specific step / operation may vary depending on the implementation method.

[0071] In step S100, the response time for a survey containing one or more items can be estimated. Step S100 is described below with reference to FIG. 8.

[0072] FIG. 8 is a flowchart specifically illustrating the step (S100) of estimating the response time for the survey of FIG. 7. Referring to FIG. 8, in step S110, the response time for each of one or more items can be estimated, and in step S120, the estimated response times for each of one or more items can be summed. Step S110 will be explained below with reference to FIG. 9.

[0073] FIG. 9 is a flowchart specifically illustrating the step (S110) of estimating the response time for each of one or more items of FIG. 8. Referring to FIG. 9, in step S111, for the first item among the one or more items, it can be determined whether the explanatory power of the first model is greater than or equal to a preset threshold. Here, the first model may correspond to a regression learning model trained to estimate the response time by receiving input items having the same type as the first item. If the explanatory power is greater than or equal to the threshold (YES), in step S112, the first item can be input into the first model and the response time can be estimated. On the other hand, if the explanatory power is less than the threshold (NO), in step S113, the response time can be estimated based on the type of the first item. Step S113 will be explained below with reference to FIG. 10.

[0074] FIG. 10 is a flowchart specifically illustrating the step (S113) of estimating response time based on the type of the first question in FIG. 9. Referring to FIG. 10, in step S113-1, it can be determined whether there are more than a pre-set number of pre-estimated first response time data for questions having the same type as the first question. If there are more than the pre-set number (YES), in step S113-2, the average of the first response time data can be estimated as the response time of the first question. If there are not more than the pre-set number (NO), in step S113-3, it can be determined whether there is a pre-set second type corresponding to the type of the first question. If there is a second type (YES), in step S113-4, the average of the pre-estimated second response time data for questions having the second type can be estimated as the response time of the first question. If Type 2 does not exist (NO), in step S113-5, the median of the previously stored total response time data can be estimated as the response time of the first question.

[0075] Returning to FIG. 7, in step S200, a score for each of the multiple evaluation items pre-set for the survey can be calculated. Embodiments related to step S200 will be described below with reference to FIG. 11 and 12.

[0076] FIG. 11 is a flowchart illustrating an example of the step (S200) of calculating the score for each of the multiple evaluation items of FIG. 7. Referring to FIG. 11, in step S210, the score for each of the first multiple evaluation items, which are pre-set for each of one or more questions, may be calculated, and in step S220, the score for each of the second multiple evaluation items, which are pre-set for the entire survey, may be calculated. For example, the first multiple evaluation items may include the length of each of one or more questions, the number of options, the number of characters in the options, and the number of options including images, and the second multiple evaluation items may include the number of open-ended questions.

[0077] FIG. 12 is a flowchart illustrating another embodiment of the step (S200) of calculating the score for each of the plurality of evaluation items of FIG. 7. Referring to FIG. 12, in step S230, a selection for a first evaluation item among the plurality of evaluation items may be received, and in step S240, only the score for each of the remaining evaluation items excluding the first evaluation item among the plurality of evaluation items may be calculated. That is, FIG. 12 may correspond to an embodiment related to excluding a specific evaluation item.

[0078] Returning to Fig. 7, in step S300, a quality score of the survey can be calculated based on the response time and the score of each of the multiple evaluation items. Then, in step S400, feedback on the survey can be output to the user based on the calculated quality score of the survey. Step S300 will be explained below with reference to Fig. 13.

[0079] FIG. 13 is a flowchart specifically illustrating the step (S300) of calculating the quality score of the survey of FIG. 7. Referring to FIG. 13, in step S310, the response time and the scores of each of the multiple evaluation items can be standardized and summed. Then, in step S320, the summed score can be subtracted from the reference score, and the result can be calculated as the quality score of the survey.

[0080] FIG. 14 is a block diagram showing the hardware configuration of a computing device (500) for survey quality evaluation according to an embodiment of the present disclosure.

[0081] Referring to FIG. 14, a computing device (500) may include one or more processors (510), a bus (530), a communication interface (540), a memory (520) for loading a computer program executed by the processor (510), and a storage (550) for storing a computer program (560). However, FIG. 14 illustrates only the components related to the embodiments of the present disclosure. Therefore, a person skilled in the art to which the present disclosure belongs will understand that other general-purpose components may be included in addition to the components illustrated in FIG. 14. That is, the computing device (500) may include various additional components in addition to the components illustrated in FIG. 14. Furthermore, depending on the case, the computing device (500) may be configured in a form in which some of the components illustrated in FIG. 14 are omitted. Hereinafter, each component of the computing device (500) will be described.

[0082] The processor (510) can control the overall operation of each component of the computing device (500). The processor (510) may be configured to include at least one of a CPU (Central Processing Unit), MPU (Micro Processor Unit), MCU (Micro Controller Unit), GPU (Graphic Processing Unit), or any form of processor well known in the art of the present disclosure. Additionally, the processor (510) may perform operations for at least one application or program for executing operations / methods according to embodiments of the present disclosure. The computing device (500) may have one or more processors.

[0083] Next, the memory (520) may store various data, commands and / or information. The memory (520) may load a computer program (560) from the storage (550) to execute an operation / method according to the embodiments of the present disclosure. The memory (520) may be implemented as a volatile memory such as RAM, but the present disclosure is not limited thereto.

[0084] Next, the bus (530) can provide communication functions between components of the computing device (500). The bus (530) can be implemented as various types of buses, such as an address bus, a data bus, and a control bus.

[0085] Next, the communication interface (540) may support wired and wireless internet communication of the computing device (500). Additionally, the communication interface (540) may support various communication methods other than internet communication. To this end, the communication interface (540) may be configured to include a communication module well known in the art of the present disclosure.

[0086] Next, the storage (550) may store one or more computer programs (560) non-temporarily. The storage (550) may be configured to include non-volatile memory such as ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), flash memory, a hard disk, a removable disk, or any form of computer-readable recording medium well known in the art to which this disclosure belongs.

[0087] Next, the computer program (560) may include one or more instructions that cause the processor (510) to perform an operation / method according to various embodiments of the present disclosure when loaded into memory (520). That is, the processor (510) may perform an operation / method according to various embodiments of the present disclosure by executing one or more loaded instructions.

[0088] For example, a computer program (560) may include instructions to perform the following actions: estimating the response time for a survey containing one or more items; calculating the score for each of a plurality of evaluation items that are pre-set for the survey; calculating the quality score of the survey based on the response time and the score for each of the plurality of evaluation items; and outputting feedback on the survey to a user based on the calculated quality score of the survey.

[0089] According to an embodiment of the present disclosure, high-quality surveys can be produced through automated survey quality evaluation, which can reduce respondent fatigue, improve the experience, and increase the likelihood of obtaining high-quality data through the survey. Furthermore, according to an embodiment of the present disclosure, the quality of the survey can be automatically evaluated in advance before the survey is completed and distributed, and by providing an interface that displays and allows for modifications to the reviewed items, the survey creator can efficiently modify the survey and prevent the need for data re-collection when responses are not properly received. Moreover, according to an embodiment of the present disclosure, the human and time costs required for survey production can be minimized by automating the survey quality inspection process.

[0090] Various embodiments of the present disclosure and effects according to the embodiments have been described with reference to FIGS. 1 to 14. The effects according to the technical concept of the present disclosure are not limited to those described above, and other unmentioned effects will be clearly understood by a person skilled in the art from the description below.

[0091] In the foregoing, although all components constituting an embodiment of the present disclosure have been described as being combined or operating together, the present disclosure is not necessarily limited to such embodiments. That is, within the scope of the purpose of the present disclosure, all components may be selectively combined and operated in one or more ways.

[0092] Although operations are depicted in a specific order in the drawings, it should not be understood that the operations must be executed in the specific order depicted or in a sequential order, or that all depicted operations must be executed to obtain the desired result. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various configurations in the embodiments described above should not be understood as a necessary separation, and it should be understood that the described program components and systems can generally be integrated together into a single software product or packaged into multiple software products.

[0093] Although embodiments of the present disclosure have been described above with reference to the attached drawings, those skilled in the art will understand that the present disclosure may be implemented in other specific forms without altering its technical concept or essential features. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. The scope of protection of the present disclosure shall be interpreted by the claims below, and all technical concepts within the equivalent scope shall be interpreted as being included within the scope of rights of the present disclosure.

Claims

Claim 1 A method for estimating survey response time performed by a computing device, comprising: a step of obtaining a survey including one or more items; a step of estimating the response time of each of the one or more items; and a step of estimating the response time for the entire survey by summing the estimated response times of each of the one or more items, wherein the step of estimating the response time of each of the one or more items comprises: a step of determining whether the explanatory power of a first model, which is trained to estimate the response time by receiving a question having the same type as the first question as input, is greater than or equal to a preset threshold for a first question among the one or more items; a step of estimating the response time of the first question by inputting the first question into the first model if the explanatory power is greater than or equal to the threshold; and a step of estimating the response time of the first question based on the type of the first question if the explanatory power is less than the threshold. Claim 2 A survey response time estimation method according to claim 1, wherein the first model is a regression analysis model trained to estimate response time through a weighted sum of scales related to the item and the item's options for each type of item, and the explanatory power of the first model is the R-squared value of the regression analysis model. Claim 3 A method for estimating survey response time, wherein, in the case where the first item is a single-type multiple-choice item, the first model is a model trained to estimate the response time of the first item based on the number of options, whether there are options including images, the number of characters in the question, and the weighted sum of the number of characters in the options. Claim 4 A survey response time estimation method according to claim 1, wherein the first model is updated at a predetermined period according to the type of item, and the explanatory power is determined together with the update. Claim 5 A method for estimating survey response time according to claim 1, wherein the step of estimating the response time of the first question based on the type of the first question comprises: a step of determining whether there exists a first response time data that has been estimated for a question having the same type as the first question, or more than a preset number; and a step of estimating the average of the first response time data as the response time of the first question if there exists more than the preset number. Claim 6 A method for estimating survey response time, wherein the first response time data is data accumulated from estimated response times for items having the same type as the first item in a survey other than the survey containing the first item. Claim 7 A method for estimating survey response time according to claim 5, wherein the step of estimating the response time of the first question based on the type of the first question further comprises: a step of determining whether a second type pre-set to correspond to the type of the first question exists when the first response time data does not exist for more than the pre-set number; and a step of estimating the average of the second response time data pre-estimated for the question having the second type as the response time of the first question when the second type exists. Claim 8 A method for estimating survey response time according to claim 7, wherein if the type of the first question is a subjective short-answer ranking type, the second type is a subjective short-answer type, and the response time of the first question is estimated by multiplying the number of options by the response time data previously estimated for the subjective short-answer question. Claim 9 A method for estimating survey response time according to claim 7, wherein the step of estimating the response time of the first question based on the type of the first question further includes the step of estimating the median of the stored total response time data as the response time of the first question when the first response time data does not exist in a number greater than the preset number and there is no second type preset to correspond to the type of the first question. Claim 10 A survey response time estimation system comprising: a processor; and a memory for storing instructions, wherein when the instructions are executed by the processor, the processor performs the operation of acquiring a survey including one or more items; the operation of estimating the response time of each of the one or more items; and the operation of estimating the response time of the entire survey by summing the estimated response times of each of the one or more items, wherein the operation of estimating the response time of each of the one or more items includes: for a first item among the one or more items, the operation of determining whether the explanatory power of a first model, which is trained to estimate the response time by receiving an item having the same type as the first item, is greater than or equal to a preset threshold; if the explanatory power is greater than or equal to the threshold, the operation of inputting the first item into the first model to estimate the response time of the first item; and if the explanatory power is less than the threshold, the operation of estimating the response time of the first item based on the type of the first item.

Images