A questionnaire sample processing method and device

Abstract

The application belongs to the technical field of computers, and particularly relates to a questionnaire sample processing method and device. The questionnaire sample processing method comprises the following steps: obtaining a questionnaire sample to be processed and user behavior data corresponding to the questionnaire sample, wherein the user behavior data comprises operation behavior of a user in the process of completing the questionnaire sample and a timestamp of a time when the operation behavior occurs; aggregating and extracting features of the user behavior data of the questionnaire sample at a preset granularity to obtain feature data of the questionnaire sample; predicting an invalid probability of the questionnaire sample based on the feature data of the questionnaire sample; and determining that the questionnaire sample is an invalid questionnaire sample when the invalid probability of the questionnaire sample is greater than a preset threshold. The questionnaire sample processing method and device based on the application can determine invalid questionnaire samples according to the invalid probability of the questionnaire samples, which is beneficial to efficiently selecting and cleaning the invalid questionnaire samples from a large number of recovered questionnaire samples, thereby improving the data reliability of the recovered questionnaire samples.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computers, and particularly relates to a questionnaire sample processing method and device. BACKGROUND

[0002] In questionnaire research, there are phenomena such as recycling or respondents not answering questions seriously, thereby affecting the reliability of a report formed by research and the accuracy of a conclusion of questionnaire research. Therefore, after questionnaire recycling is completed, the invalid samples can be removed by a way of manually judging the effectiveness of the questionnaire. However, if the invalid samples are determined by the way of manually judging the effectiveness of the questionnaire, a large amount of human and time costs are required, the judgment speed is low, and each operator needs to have high sensitivity to invalid samples to accurately identify the invalid samples, and the operability is low.

[0003] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present application, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0004] The purpose of the present application is to provide a questionnaire sample processing method and device, which at least partially overcomes the low recognition efficiency of invalid samples in the related art by using a way of manually judging the effectiveness of the questionnaire.

[0005] Other characteristics and advantages of the present application will become apparent from the following detailed description, or will be learned by practice of the present application.

[0006] According to an aspect of an embodiment of the present application, a questionnaire sample processing method is provided, and the method comprises:

[0007] Obtaining a questionnaire sample to be processed and user behavior data corresponding to the questionnaire sample, wherein the user behavior data comprises an operation behavior of a user in completing the questionnaire sample and a time stamp of a time when the operation behavior occurs;

[0008] Aggregating and extracting features of the user behavior data of the questionnaire sample at a preset granularity to obtain feature data of the questionnaire sample;

[0009] Predicting an invalid probability of the questionnaire sample based on the feature data of the questionnaire sample;

[0010] When the invalid probability of the questionnaire sample is greater than a preset threshold, determining that the questionnaire sample is an invalid questionnaire sample.

[0011] According to an aspect of an embodiment of the present application, a questionnaire sample processing device is provided, and the questionnaire sample processing device comprises:

[0012] a sample data obtaining module configured to obtain a questionnaire sample to be processed and user behavior data corresponding to the questionnaire sample, wherein the user behavior data comprises operation behavior of a user in completing the questionnaire sample and a timestamp of a time when the operation behavior occurs;

[0013] a feature data obtaining module configured to aggregate and extract features of the user behavior data of the questionnaire sample at a preset granularity to obtain feature data of the questionnaire sample;

[0014] an invalid probability predicting module configured to predict an invalid probability of the questionnaire sample based on the feature data of the questionnaire sample;

[0015] a questionnaire sample determining module configured to determine that the questionnaire sample is an invalid questionnaire sample when the invalid probability of the questionnaire sample is greater than a preset threshold.

[0016] In some embodiments of the present application, based on the above technical solutions, the feature data obtaining module comprises:

[0017] an intermediate data obtaining unit configured to aggregate and extract features of the user behavior data of the questionnaire sample at a question granularity to obtain intermediate data at a question granularity;

[0018] a feature data obtaining unit configured to aggregate and extract features of the intermediate data at a questionnaire sample granularity to obtain the feature data of the questionnaire sample.

[0019] In some embodiments of the present application, based on the above technical solutions, the invalid probability predicting module comprises:

[0020] a feature data dividing unit configured to divide the feature data into numerical feature data and non-numerical feature data;

[0021] a normalization processing unit configured to perform normalization processing on the numerical feature data to obtain first preprocessed data;

[0022] a numerical processing unit configured to perform numerical processing on the non-numerical data to obtain second preprocessed data;

[0023] an invalid probability predicting unit configured to input the first preprocessed data and the second preprocessed data into a questionnaire sample cleaning model composed of one or more classifiers to obtain the invalid probability of the questionnaire sample, wherein the questionnaire sample cleaning model is used to predict the invalid probability of the questionnaire sample.

[0024] In some embodiments of the present application, based on the above technical solutions, the questionnaire sample processing apparatus further comprises:

[0025] a training set establishing unit configured to obtain questionnaire samples marked with result labels and user behavior data corresponding to the questionnaire samples to establish a training set, wherein the result labels include valid labels and invalid labels;

[0026] a classifier training unit configured to train one or more classifiers using the training set;

[0027] a cleaning model constructing unit configured to construct a questionnaire sample cleaning model according to the one or more trained classifiers.

[0028] In some embodiments of the present application, based on the above technical solutions, the cleaning model constructing unit comprises:

[0029] a training effect obtaining sub-unit configured to obtain classification accuracy, classification precision and questionnaire sample recall rate of the one or more trained classifiers;

[0030] a cleaning model combining sub-unit configured to select one or more of the classifiers to combine into the questionnaire sample cleaning model according to the classification accuracy, the classification precision and the classification recall rate of the classifiers.

[0031] In some embodiments of the present application, based on the above technical solutions, the training set establishing unit comprises:

[0032] an invalid questionnaire sample obtaining sub-unit configured to obtain invalid questionnaire samples marked with invalid labels and add the invalid questionnaire samples to the training set;

[0033] a mislabeled questionnaire sample obtaining sub-unit configured to obtain mislabeled questionnaire samples marked with valid labels and add the mislabeled questionnaire samples to the training set, the mislabeled questionnaire samples being questionnaire samples misjudged as invalid questionnaire samples by the questionnaire sample cleaning model;

[0034] a valid questionnaire sample obtaining sub-unit configured to obtain valid questionnaire samples marked with valid labels and add the valid questionnaire samples to the training set, so that a proportion of the invalid questionnaire samples in all questionnaire samples with result labels in the training set is within a preset proportion range.

[0035] In some embodiments of the present application, based on the above technical solutions, the sample data obtaining module comprises:

[0036] a data real-time grabbing unit configured to grab real-time uploaded user behavior data, questionnaire samples and questionnaire topic data, wherein the user behavior data is bound with questionnaire labels and sample labels, the questionnaire samples are bound with questionnaire labels and sample labels, and the questionnaire topic data is bound with questionnaire labels and sample labels.

[0037] a data aggregation unit configured to aggregate the user behavior data, the questionnaire sample and the questionnaire question data bound with the same questionnaire label and the same sample label, to obtain a questionnaire sample to be processed, user behavior data corresponding to the questionnaire sample and question data corresponding to the questionnaire sample.

[0038] In some embodiments of the present application, based on the above technical solution, the invalid questionnaire sample includes a high-probability invalid questionnaire sample and a suspected invalid questionnaire sample; and the questionnaire sample determination module includes:

[0039] a suspected invalid determination unit configured to determine the questionnaire sample as a suspected invalid questionnaire sample when the invalid probability of the questionnaire sample is greater than a first preset threshold and less than or equal to a second preset threshold;

[0040] a high-probability invalid determination unit configured to determine the questionnaire sample as a high-probability invalid questionnaire sample when the invalid probability of the questionnaire sample is greater than the second preset threshold.

[0041] In some embodiments of the present application, based on the above technical solution, the questionnaire sample processing apparatus further includes:

[0042] a questionnaire sample recycling module configured to recycle questionnaire samples from one or more channels;

[0043] an invalid probability acquisition module configured to acquire the invalid probability of the questionnaire sample in each of the channels respectively;

[0044] a channel quality diagnosis module configured to diagnose the channel quality of each of the channels according to the distribution of the invalid probability of the questionnaire sample corresponding to each of the channels.

[0045] According to an aspect of an embodiment of the present application, there is provided a computer readable medium having stored thereon a computer program, which, when executed by a processor, implements the questionnaire sample processing method in the above technical solution.

[0046] According to an aspect of an embodiment of the present application, there is provided an electronic device, including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the questionnaire sample processing method in the above technical solution via execution of the executable instructions.

[0047] According to one aspect of the embodiments of this application, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the questionnaire sample processing method as described in the above technical solutions.

[0048] In the technical solution provided in this application embodiment, the user behavior data of the questionnaire sample is aggregated and feature extracted at a preset granularity to obtain the feature data of the questionnaire sample. Then, the invalidity probability of the questionnaire sample is predicted based on the feature data of the questionnaire sample. When the invalidity probability of the questionnaire sample is greater than a preset threshold, the questionnaire sample is determined to be an invalid questionnaire sample. This realizes the prediction of the invalidity probability of the questionnaire sample and can determine the invalid questionnaire sample based on the invalidity probability. This is beneficial to efficiently select and clean invalid questionnaire samples from a large number of collected questionnaire samples, thereby improving the data reliability of the collected questionnaire samples.

[0049] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0050] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0051] Figure 1 An exemplary system architecture block diagram illustrating the application of the technical solution of this application is shown schematically.

[0052] Figure 2 A flowchart illustrating the steps of a questionnaire sample processing method according to certain embodiments of this application is shown schematically.

[0053] Figure 3 This illustration shows a visualization of how certain embodiments of this application process questionnaire samples and corresponding user behavior data, and determine invalid questionnaire samples.

[0054] Figure 4 This illustration shows a page displaying the summary results of invalid questionnaire samples from a certain embodiment of this application.

[0055] Figure 5 The illustration schematically shows a monthly monitoring chart after classifying questionnaire sample data according to a certain embodiment of this application.

[0056] Figure 6 A flowchart schematically shows a step of obtaining a questionnaire sample to be processed and user behavior data corresponding to the questionnaire sample in some embodiments of the present application.

[0057] Figure 7 A flowchart schematically shows a step of aggregating and extracting features of user behavior data of a questionnaire sample at a preset granularity to obtain feature data of the questionnaire sample in some embodiments of the present application.

[0058] Figure 8 A process diagram schematically shows a step of aggregating and extracting features of user behavior data of a questionnaire sample at a preset granularity to obtain feature data of the questionnaire sample in some embodiments of the present application.

[0059] Figure 9 A flowchart schematically shows a step of predicting an invalid probability of a questionnaire sample based on feature data of the questionnaire sample in some embodiments of the present application.

[0060] Figure 10 A process diagram schematically shows a step of predicting an invalid probability of a questionnaire sample based on feature data of the questionnaire sample in some embodiments of the present application.

[0061] Figure 11 A flowchart schematically shows a step of inputting first preprocessed data and second preprocessed data into a questionnaire sample cleaning model composed of one or more classifiers to obtain an invalid probability of a questionnaire sample in some embodiments of the present application.

[0062] Figure 12 A flowchart schematically shows a step of constructing a questionnaire sample cleaning model according to one or more classifiers trained in some embodiments of the present application.

[0063] Figure 13 A process diagram schematically shows a step of constructing a training set, model training and selection, model online application, and new data generation in some embodiments of the present application.

[0064] Figure 14 A flowchart schematically shows a step of obtaining a questionnaire sample labeled with a result tag and user behavior data corresponding to the questionnaire sample to establish a training set in some embodiments of the present application.

[0065] Figure 15 A flowchart schematically shows a step of determining a questionnaire sample as an invalid questionnaire sample when an invalid probability of the questionnaire sample is greater than a preset threshold in some embodiments of the present application.

[0066] Figure 16 A flowchart schematically shows a step of determining a questionnaire sample as an invalid questionnaire sample when an invalid probability of the questionnaire sample is greater than a preset threshold in some embodiments of the present application.

[0067] Figure 17a A distribution diagram of the invalid probability of the questionnaire sample corresponding to the first channel in some embodiments of the present application is schematically shown.

[0068] Figure 17b A distribution diagram of the invalid probability of the questionnaire sample corresponding to the second channel in some embodiments of the present application is schematically shown.

[0069] Figure 17c A distribution diagram of the invalid probability of the questionnaire sample corresponding to the third channel in some embodiments of the present application is schematically shown.

[0070] Figure 18 A structure block diagram of the questionnaire sample processing device provided by the embodiments of the present application is schematically shown.

[0071] Figure 19 A computer system structure block diagram of the electronic device for implementing the embodiments of the present application is schematically shown. DETAILED DESCRIPTION

[0072] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations can be implemented in any

[0073] Moreover, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the application. One skilled in the relevant art will recognize, however, that the

[0074] The block diagrams in the drawings show only the functional entities and do not necessarily have to correspond to physically independent entities. That is, the functional entities can be implemented in software, or in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0075] The flowcharts shown in the drawings are only exemplary illustrations and do not necessarily have to include all contents and operations / steps, nor do they have to be executed in the order described. For example, some operations / steps can be further broken down, and some operations / steps can be combined or partially combined, so that the actual execution order can be changed according to the actual situation.

[0076] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on cloud computing business model application, which can form a resource pool and be used on demand. Cloud computing technology will become an important support. The background service of the technical network system needs a large amount of computing and storage resources, such as video websites, picture websites and more portals. With the high development and application of the Internet industry, every item may have its own identification mark in the future, and it needs to be transmitted to the background system for logical processing. Different levels of data will be processed separately, and various industry data will need strong system support, which can only be realized through cloud computing.

[0077] Artificial intelligence cloud service, also known as AIaaS (AI as a Service), is a common service mode of artificial intelligence platform. Specifically, the AIaaS platform will split several common AI services and provide independent or packaged services on the cloud. This service mode is similar to opening an AI theme mall: all developers can access one or more artificial intelligence services provided by the platform through API interface, and some experienced developers can also use the AI framework and AI infrastructure provided by the platform to deploy and maintain their own cloud artificial intelligence services.

[0078] Database, in short, can be regarded as an electronic file cabinet - a place to store electronic files, users can add, query, update, delete and other operations on the data in the file. The so-called "database" is a collection of data stored together in a certain way, shared by multiple users, with as little redundancy as possible, and independent of application programs.

[0079] Database management system (English: Database Management System, DBMS for short) is a computer software system designed to manage databases, generally with basic functions such as storage, interception, security, backup, etc. Database management systems can be classified according to the database model they support, such as relational, XML (Extensible Markup Language, XML); or according to the type of computer they support, such as server clusters, mobile phones; or according to the query language they use, such as SQL (Structured Query Language, SQL), XQuery; or according to the performance focus, such as maximum size, highest speed; or other classification methods. Regardless of the classification method used, some DBMS can cross categories, such as supporting multiple query languages at the same time.

[0080] Blockchain is a new application mode of distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm and other computer technologies. Blockchain, in essence, is a decentralized database, which is a chain of data blocks associated using cryptographic methods, each data block containing information of a batch of network transactions, used to verify the validity of the information (anti-fake) and generate the next block. The blockchain can include a blockchain underlying platform, a platform product service layer, and an application service layer.

[0081] The blockchain underlying platform can include user management, basic services, smart contracts, and operation monitoring processing modules. Among them, the user management module is responsible for the identity information management of all blockchain participants, including maintaining the generation of public and private keys (account management), key management, and the maintenance of the correspondence between the real identity and the blockchain address (permission management), etc., and under authorization, supervising and auditing the transaction of certain real identities, providing rule configuration for risk control (risk audit); the basic service module is deployed on all blockchain node devices to verify the validity of business requests, and after consensus, the valid requests are recorded on the storage. For a new business request, the basic service first parses and authenticates the interface (interface adaptation), then encrypts the business information through a consensus algorithm (consensus management), and after encryption, the complete and consistent transmission is transmitted to the shared ledger (network communication) and recorded and stored; the smart contract module is responsible for the registration and issuance of contracts, as well as the triggering and execution of contracts. Developers can define contract logic through a certain programming language, publish it to the blockchain (contract registration), call keys or other events to trigger execution according to the logic of the contract terms, complete the contract logic, and also provide functions for contract upgrade and cancellation; the operation monitoring module is mainly responsible for the deployment, configuration modification, contract setting, cloud adaptation in the product release process, and the real-time state visualization output in the product running, such as alarm, monitoring network conditions, monitoring node device health status, etc.

[0082] The platform product service layer provides basic capabilities and implementation frameworks for typical applications. Developers can stack business features based on these basic capabilities to complete the blockchain implementation of business logic. The application service layer provides application services based on the blockchain solution for business participants to use.

[0083] It should be noted that the data involved in the present application can be stored in the blockchain, such as the characteristic data of the questionnaire sample.

[0084] Figure 1 An exemplary system architecture block diagram to which the technical solutions of the present application are applied is schematically shown.

[0085] As Figure 1As shown, the system architecture 100 can include a terminal device 110, a network 120 and a server 130. The terminal device 110 can include various electronic devices such as a smart phone, a tablet computer, a notebook computer, a desktop computer, etc. The server 130 can be a stand-alone physical server, or a server cluster composed of multiple physical servers or a distributed system, or a cloud server providing cloud computing services. The network 120 can be various connection types of communication media capable of providing a communication link between the terminal device 110 and the server 130, for example, can be a wired communication link or a wireless communication link.

[0086] According to the implementation needs, the system architecture in the embodiments of the present application can have any number of terminal devices, networks and servers. For example, the server 130 can be a server group composed of multiple server devices. In addition, the technical solutions provided by the embodiments of the present application can be applied to the terminal device 110, or can be applied to the server 130, or can be jointly implemented by the terminal device 110 and the server 130, which is not specially limited by the present application.

[0087] For example, the server 130 can be loaded with the questionnaire sample processing method of the embodiments of the present application. After the user fills out the questionnaire, the recovered questionnaire sample can be uploaded to the server 130 in real time through the client 110, and then the questionnaire sample processing method of the embodiments of the present application is implemented by the server 130 to eliminate invalid samples and realize data cleaning of the questionnaire samples. Therefore, the invalid probability of the questionnaire sample can be predicted, and the invalid questionnaire sample can be determined according to the invalid probability of the questionnaire sample, which is beneficial to efficiently select and clean the invalid questionnaire sample from a large number of recovered questionnaire samples, thereby improving the data reliability of the recovered questionnaire samples.

[0088] The server can be a stand-alone physical server, or a server cluster composed of multiple physical servers or a distributed system, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and basic cloud computing services such as big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted device, etc., but is not limited thereto. The terminal and the server can be directly or indirectly connected through wired or wireless communication, which is not limited by the present application.

[0089] It can be understood that the manual sample cleaning based on the rule setting needs to be manually cleaned by manual operation, which is time-consuming and laborious, and needs to have professional researchers to set the rejection rules or trap questions, and the operation personnel are required to be higher, and the universality is low, and the questionnaire sample processing method of the embodiments of the application can determine the invalid questionnaire sample according to the invalid probability of the questionnaire sample, which is beneficial to efficiently select and clean the invalid questionnaire sample from the large amount of questionnaire samples recovered, so as to determine the invalid questionnaire sample at a lower time cost and labor cost, thereby realizing data cleaning, and the universality is higher.

[0090] The questionnaire sample processing method provided by the application will be described in detail in combination with specific embodiments.

[0091] Figure 2 The step flowchart of the questionnaire sample processing method of some embodiments of the application is schematically shown. The execution subject of the questionnaire sample processing method can be a terminal device or a server, etc., which is not limited in the application. As shown in Figure 2 The questionnaire sample processing method mainly can include the following steps S210-S240.

[0092] S210. Obtain the questionnaire sample to be processed and the user behavior data corresponding to the questionnaire sample, wherein the user behavior data includes the operation behavior of the user in the process of completing the questionnaire sample and the time stamp of the time when the operation behavior occurs;

[0093] S220. Aggregate and feature extract the user behavior data of the questionnaire sample at a preset granularity to obtain the feature data of the questionnaire sample;

[0094] S230. Predict the invalid probability of the questionnaire sample based on the feature data of the questionnaire sample;

[0095] S240. When the invalid probability of the questionnaire sample is greater than a preset threshold, determine that the questionnaire sample is an invalid questionnaire sample.

[0096] The questionnaire is published by the questionnaire master, and after the user fills in the questionnaire to generate the questionnaire sample and submits the questionnaire sample, the questionnaire sample to be processed and the user behavior data corresponding to the questionnaire sample can be obtained from the database of real-time entry of the questionnaire sample, the message queue of real-time entry of the questionnaire sample or the database storing the historical questionnaire data. The questionnaire sample is the questionnaire and the corresponding related filling data formed after the user fills in and submits the questionnaire, that is, the questionnaire and the corresponding related filling data recovered by the database. The user behavior data includes the operation behavior of the user in the process of completing the questionnaire sample and the time stamp of the time when the operation behavior occurs. Specifically, the time stamp can be accurate to milliseconds.

[0097] Figure 3A visual diagram of processing questionnaire samples and corresponding user behavior data of the questionnaire samples by certain embodiments of the present application, and determining invalid questionnaire samples is shown. As shown in Figure 3 the horizontal axis is a timestamp, each row represents a recovered questionnaire sample, each questionnaire sample has a corresponding unique questionnaire sample number, and each dot in each row represents a user behavior data of a user. The user behavior data can include one or more of the following data: opening a questionnaire, moving a mouse, clicking a mouse, scrolling a mouse, turning a page, filling in a question, and submitting a questionnaire.

[0098] The user behavior data of the questionnaire samples is aggregated and feature extracted at a preset granularity to obtain feature data of the questionnaire samples. The feature data can include one or more of the following data: questionnaire answering time, question answering time, question reading time, total number of answers, question content length, number of question options, question answer options, user reading speed, user source, and user device. Different feature data can have different correlation coefficients with the invalidity of the questionnaire samples.

[0099] Then, the invalid probability of the questionnaire samples is predicted based on the feature data of the questionnaire samples, and when the invalid probability of the questionnaire samples is greater than a preset threshold, the questionnaire samples are determined to be invalid questionnaire samples. The determination results of the multiple questionnaire samples are visualized, and as shown in Figure 3 the questionnaire samples in the box are invalid questionnaire samples.

[0100] The determination of the questionnaire samples as invalid samples means that the questionnaire is determined to be a sample that affects the final conclusion in the questionnaire survey. This part of the sample is likely to come from a user who does not seriously answer the question or a questionnaire sample recovery. After determining the questionnaire samples as invalid questionnaire samples, the invalid questionnaire samples can be summarized and the summary results can be displayed to the questionnaire owner. Figure 4 A display page of the summary results of the invalid questionnaire samples summarized by certain embodiments of the present application is shown. As shown in Figure 4As shown, the invalid questionnaire samples are displayed and arranged. The "username" is the username of the user who filled in the invalid questionnaire sample. The "answering duration" is the duration that the user of the invalid questionnaire sample spent from opening the questionnaire to submitting the questionnaire. The "red packet reward" is the red packet distribution of the questionnaire. Some platforms or questionnaire owners may encourage users to fill in questionnaires by distributing red packets, which may increase the probability of obtaining invalid questionnaire samples. The "auditor" is the person who determines the invalid questionnaire sample. If the questionnaire sample processing method of the embodiments of the present application determines the invalid questionnaire sample, the auditor is "intelligent recognition". If the auditor staff determines the invalid questionnaire sample in the questionnaire sample by manual operation, the specific auditor staff is marked, such as "Lynnliu". Figure 4 The summary result display page of the invalid questionnaire sample also provides a viewing entrance of the questionnaire, and the answering situation and user behavior data of the questionnaire sample corresponding to the row can be viewed by clicking "click to view". The "submission time" is the submission date and time of the invalid questionnaire sample. Clicking "click to mark as valid questionnaire sample" in the "operation" column can reverse the invalid questionnaire sample to a valid questionnaire sample. Thus, the questionnaire owner can view and operate the questionnaire sample by clicking the "click to view" in the "operation" column, and the questionnaire owner can reverse the invalid questionnaire sample to a valid questionnaire sample by clicking "click to mark as valid questionnaire sample" in the "operation" column. Figure 4 The summary result display page of the invalid questionnaire sample reverses the misjudged invalid questionnaire sample, which is beneficial to reduce the misjudgment rate of the questionnaire sample processing method of some embodiments of the present application and improve the accuracy of the judgment of the questionnaire sample. When the questionnaire owner finds that too many valid questionnaire samples are misjudged as invalid questionnaire samples, the questionnaire owner can click "too many mislabeled questionnaires? Click to feedback." to feedback that the misjudgment rate is too high, which is beneficial to the questionnaire sample processing method of some embodiments of the present application to modify the operation in steps S220, S230 or S240 according to the feedback, and is also beneficial to the further training and evolution of the questionnaire sample cleaning model to improve the accuracy of the judgment of the invalid probability of the questionnaire sample.

[0101] After the above operation, the monthly monitoring chart after the questionnaire sample data is classified can be obtained as shown. Figure 5 As shown, the horizontal axis is the generation time of the questionnaire sample, and the vertical axis is the number of questionnaire samples. As shown, Figure 5 As shown, the horizontal axis is the generation time of the questionnaire sample, and the vertical axis is the number of questionnaire samples. As shown, Figure 5As shown in the middle, all the questionnaire samples collected in each month can be classified into valid questionnaire samples, manually marked invalid questionnaire samples, automatically determined invalid questionnaire samples, and counter-labeled questionnaire samples. Among them, the manually marked invalid questionnaire samples are determined by the auditing staff through manual determination of the invalid questionnaire samples in the questionnaire samples to obtain the determination results of the invalid questionnaire samples. The automatically determined invalid questionnaire samples are the questionnaire samples determined as invalid questionnaire samples by the questionnaire processing method of the embodiment of the application. The valid questionnaire samples are the questionnaire samples audited as valid by the auditing staff, or the questionnaire samples not determined as invalid questionnaire samples by the questionnaire processing method of the embodiment of the application. The counter-labeled questionnaire samples are the questionnaire samples misjudged as invalid questionnaire samples by the questionnaire sample cleaning model, and then actively counter-labeled as valid questionnaire samples by manual. Thus, all the questionnaire samples collected in each month can be classified into valid questionnaire samples, manually marked invalid questionnaire samples, automatically determined invalid questionnaire samples, and counter-labeled questionnaire samples, and a column chart or other visual chart can be generated and displayed to the questionnaire owner, so that the questionnaire owner can intuitively analyze the components of the questionnaire, thereby better grasping the questionnaire collection situation and taking corresponding measures such as increasing questionnaire distribution channels and canceling red packet incentives.

[0102] In some embodiments, after determining that the questionnaire sample is an invalid questionnaire sample, the invalid questionnaire sample can be stored in a preset database for subsequent analysis and summary of the invalid questionnaire sample. The questionnaire samples from which the invalid questionnaire samples are removed can also be stored in another preset database, so that subsequent data analysis of the questionnaire survey questions can be performed based on the questionnaire samples from which the invalid questionnaire samples are removed, and a questionnaire report can be output, thereby improving the accuracy of the output report and conclusion of the questionnaire survey.

[0103] Figure 6 The step flowchart of acquiring the questionnaire samples to be processed and the user behavior data corresponding to the questionnaire samples in some embodiments of the application is schematically shown. As shown in Figure 6 Based on the above embodiments, the acquisition of the questionnaire samples to be processed and the user behavior data corresponding to the questionnaire samples in step S210 can further include steps S610-S620.

[0104] S610. Grab the real-time uploaded user behavior data, questionnaire samples and questionnaire topic data, wherein the user behavior data is bound with the questionnaire label and the sample label, the questionnaire sample is bound with the questionnaire label and the sample label, and the questionnaire topic data is bound with the questionnaire label and the sample label;

[0105] S620. Aggregate the user behavior data, the questionnaire samples and the questionnaire topic data bound with the same questionnaire label and the same sample label to obtain the questionnaire samples to be processed, the user behavior data corresponding to the questionnaire samples and the topic data corresponding to the questionnaire samples.

[0106] The raw data of the user behavior data can be stored in the database Elasticsearch or the columnar database Clickhouse through an interface after raw data collection. Elasticsearch is a distributed search server. Clickhouse is a columnar storage database. When a new questionnaire sample is collected, the questionnaire sample can be added to the Kafka message queue and stored in the questionnaire collection database through the message queue. The questionnaire background database stores all the questions and related settings of the questionnaire created by the questionnaire master.

[0107] When the questionnaire sample to be processed and the user behavior data corresponding to the questionnaire sample are acquired, the user behavior data can be pulled from the user behavior database in real time in an asynchronous manner, all the question data of the related questionnaire, settings, etc. are pulled from the questionnaire background database, and the related questionnaire sample collected from the questionnaire collection database. Then, the user behavior data, the questionnaire sample and the questionnaire question data bound with the same questionnaire label and the same sample label are aggregated to obtain the questionnaire sample to be processed, the user behavior data corresponding to the questionnaire sample and the question data corresponding to the questionnaire sample. Thus, the real-time and rapid pulling of data can be realized by aggregating the user behavior data, the questionnaire sample and the questionnaire question data bound with the same questionnaire label and the same sample label, so that the questionnaire sample processing method of the present application can be used for sample cleaning of the real-time uploaded user behavior data, the questionnaire sample and the questionnaire question data, which is beneficial to real-time monitoring of the invalid questionnaire sample proportion of the questionnaire sample of the questionnaire collection big board.

[0108] Figure 7 A step flowchart of aggregating and feature extracting the user behavior data of the questionnaire sample in a preset granularity to obtain the feature data of the questionnaire sample in some embodiments of the present application is schematically shown. Figure 7 As shown in the above embodiment, the aggregation and feature extraction of the user behavior data of the questionnaire sample in a preset granularity to obtain the feature data of the questionnaire sample in step S220 can further include the following steps S710-S720.

[0109] S710. The user behavior data of the questionnaire sample is aggregated and feature extracted in a question granularity to obtain intermediate data in a question granularity;

[0110] S720. The intermediate data is aggregated and feature extracted in a questionnaire sample granularity to obtain the feature data of the questionnaire sample.

[0111] In some embodiments, the preset granularity can include a question granularity and a questionnaire sample granularity.

[0112] Figure 8A process diagram of aggregating and extracting features of user behavior data of a questionnaire sample in a preset granularity in some embodiments of the present application is shown. As shown in Figure 8 The user behavior data is pulled from the user behavior database first to obtain the user behavior data shown in the dashed box 810. The questionnaire sample label of the user behavior data is 021, the user identity label of the user behavior data is ly1277, and the questionnaire label of the user behavior data is 001. In the user behavior data shown in the dashed box 810, it is shown that at the time stamp 0s, the user opens the questionnaire; at the time stamp 0.5s, the user fills in question A; at the time stamp 3s, the user fills in question A again; at the time stamp 4s, the user fills in question A for the third time; at the time stamp 6s, the user fills in question B; at the time stamp 9s, the user fills in question B again; and at the time stamp 10s, the user submits the questionnaire.

[0113] Then, all the questions, settings and question data of the related questionnaire are pulled from the questionnaire background database, and the related questionnaire sample is pulled from the questionnaire recycling database. The user behavior data, the questionnaire sample and the questionnaire question data bound with the same questionnaire label and the same sample label are aggregated, and the user behavior data of the questionnaire sample is aggregated and extracted in the question granularity to obtain the intermediate data in the question granularity shown in the dashed box 820 in Figure 8 The intermediate data in the question granularity shown in the dashed box 820 in includes two groups of data in the question granularity.

[0114] The first group of data includes the data aggregated and extracted around the question label question A: according to the time stamp 0s when the user opens the questionnaire and the time stamp 0.5s when the user fills in question A for the first time in the user behavior data, the reading time of question A is 0.5s; according to the time stamp 0s when the user opens the questionnaire and the time stamp 4s when the user fills in question A for the third time in the user behavior data, the answering time of question A is 4s; according to the number of times of filling in question A in the user behavior data, the number of times of answering question A is 3; according to the questionnaire question data pulled from the questionnaire question database, the content length of question A is 16 and the number of options is 3. The extraction process of other features in the question granularity is similar.

[0115] The second set of data includes aggregation and feature extraction around the question labeled Question B: Based on the user behavior data, the reading time for Question B is 2 seconds, obtained from the timestamp of the user completing Question A for the third time (4 seconds) and the timestamp of the user completing Question B for the first time (6 seconds); the answering time for Question B is 5 seconds, obtained from the timestamp of the user completing Question A for the third time (4 seconds) and the timestamp of the user completing Question B for the second time (9 seconds); the number of times Question B is answered is 2, obtained from the number of times Question B is answered in the user behavior data; and the content length of Question B is 10 and the number of options is 1, obtained from the questionnaire question data pulled from the questionnaire question database (when the number of options is 1, the question can be a fill-in-the-blank question or a non-mandatory question). The extraction process of other features at the question level is similar.

[0116] Then, the intermediate data is aggregated and features are extracted at the questionnaire sample granularity to obtain the questionnaire sample as follows: Figure 8 The feature data is shown in the dashed box 830. Figure 8 The specific process of aggregation and feature extraction of the feature data of the questionnaire sample shown in the dashed box 830 is as follows: Based on the user behavior data, the timestamp of the user opening the questionnaire (0s) and the timestamp of the user submitting the questionnaire (10s) are used to obtain the total answering time of the questionnaire (10s); the reading ratio of (0,1)s is the proportion of the number of questions indicated by the question-filling behavior with timestamps between (0,1)s to all questions. In this questionnaire sample, the question-filling behavior with timestamps between (0,1)s indicates only question A. The number of questions with timestamps of 1 accounts for 0.5 of the total number of questions 2. Therefore, the reading ratio of (0,1)s is 0.5; the answering ratio of (0,1)s is the proportion of the number of questions indicated by the question-filling behavior with timestamps between (0,1)s to all questions. In this questionnaire sample, the number of questions-filling behavior with timestamps between (0,1)s to all questions is 0.5. The action of completing the question is not present, and the number of questions indicated by the action of completing the question is 0. Therefore, the answer ratio of (0,1)s is 0. The questions include question A and question B, and the total number of questions is 2. The source can be recorded in the questionnaire collection database. The channel source of the questionnaire sample collection is WeChat. The user device can be recorded in the questionnaire collection database. In this questionnaire sample, the user device that generated and sent the questionnaire sample is an iOS device. According to the answering time of question A and question B, the sum of the answering time of question A and question B is 4s + 5s = 9s. According to the content length of question A and question B, the sum of the content length of question A and question B is 26. According to the sum of the answering time of question A and question B, 9s, and the sum of the content length of question A and question B, 26, the average reading speed of this questionnaire sample is 26 / 9 = 2.89.

[0117] In some embodiments, the user behavior data that cannot be further aggregated and feature extracted can be directly output as intermediate data; the intermediate data that cannot be further aggregated and feature extracted can be directly output as feature data.

[0118] Thus, by aggregating and feature extracting the user behavior data of the questionnaire sample at the question granularity to obtain intermediate data at the question granularity, and then aggregating and feature extracting the intermediate data at the questionnaire sample granularity to obtain the feature data of the questionnaire sample, the questionnaire sample is aggregated and feature extracted at the preset granularity to obtain the feature data of the questionnaire sample, thereby facilitating subsequent processing based on the feature data of the questionnaire sample, and finally obtaining the invalid probability of the questionnaire sample and the determination result of whether the questionnaire sample is an invalid questionnaire sample.

[0119] In other embodiments, the preset granularity can also include other granularities. For example, a page turning granularity, etc. The user behavior data of the questionnaire sample can be aggregated and feature extracted at the page turning granularity to obtain intermediate data at the page turning granularity, and then the intermediate data can be aggregated and feature extracted at the questionnaire sample granularity to obtain the feature data of the questionnaire sample. For example, the user behavior data related to page turning is aggregated and feature extracted to obtain the page turning speed and the number of page turns. Thus, the user behavior data of the questionnaire sample can be aggregated and feature extracted in multiple ways, which is beneficial to improve the accuracy of subsequent judgment of the invalid probability of the questionnaire sample.

[0120] Figure 9 A step flowchart of predicting the invalid probability of the questionnaire sample based on the feature data of the questionnaire sample in some embodiments of the present application is schematically shown. As shown in Figure 9 Based on the above embodiments, the step of predicting the invalid probability of the questionnaire sample based on the feature data of the questionnaire sample in step S230 can further include steps S910-S940.

[0121] S910. Dividing the feature data into numerical class feature data and non-numerical class feature data;

[0122] S920. Normalizing the numerical class feature data to obtain first preprocessed data;

[0123] S930. Numericalizing the non-numerical class data to obtain second preprocessed data;

[0124] S940. Inputting the first preprocessed data and the second preprocessed data into a questionnaire sample cleaning model composed of one or more classifiers to obtain the invalid probability of the questionnaire sample, wherein the questionnaire sample cleaning model is used to predict the invalid probability of the questionnaire sample.

[0125] For example, the numerical feature data can include total answering time of the questionnaire, content length of the question, number of options, proportion of question reading time, proportion of question answering time, and total number of answered questions. The non-numerical data can include user source and user equipment, etc.

[0126] Figure 10 A process diagram for predicting the invalid probability of a questionnaire sample based on feature data of the questionnaire sample in some embodiments of the present application is shown. Please refer to Figure 10 , the feature data of the questionnaire sample is first obtained, and then the feature data of the questionnaire sample is pre-processed and input into a questionnaire sample cleaning model to obtain the invalid probability of the questionnaire sample. Then, according to a pre-configured preset threshold, when the invalid probability of the questionnaire sample is greater than the preset threshold, the questionnaire sample is marked as an invalid questionnaire sample, and the invalid questionnaire sample is reported for monitoring. If the invalid probability of the questionnaire sample is less than or equal to the preset threshold, the questionnaire sample can be marked as a valid questionnaire sample, or the questionnaire sample can not be marked. The data preprocessing includes numerical processing or normalization processing. Specifically, the feature data can be divided into numerical feature data and non-numerical feature data, then the numerical feature data is normalized to obtain first pre-processing data, and the non-numerical data is numerically processed to obtain second pre-processing data. Then the first pre-processing data and the second pre-processing data are input into a questionnaire sample cleaning model composed of one or more classifiers to obtain the invalid probability of the questionnaire sample, wherein the questionnaire sample cleaning model is used to predict the invalid probability of the questionnaire sample. The questionnaire sample cleaning model can be an artificial intelligence model.

[0127] Figure 11 A step flowchart is schematically shown before the first pre-processing data and the second pre-processing data are input into the questionnaire sample cleaning model composed of one or more classifiers to obtain the invalid probability of the questionnaire sample in some embodiments of the present application. As Figure 11 shown, on the basis of the above embodiments, before the first pre-processing data and the second pre-processing data are input into the questionnaire sample cleaning model composed of one or more classifiers to obtain the invalid probability of the questionnaire sample in step S940, the following steps S1110-S1130 can be further included.

[0128] S1110. Obtain questionnaire samples marked with result labels and user behavior data corresponding to the questionnaire samples to establish a training set, wherein the result labels include valid labels and invalid labels;

[0129] S1120. Train one or more classifiers using the training set;

[0130] S1130. Construct a questionnaire sample cleaning model according to the trained one or more classifiers.

[0131] Specifically, the questionnaire sample marked with the valid label is considered as a valid questionnaire sample; the questionnaire sample marked with the invalid label is considered as an invalid questionnaire sample. The training set includes the questionnaire sample marked with the valid label and the questionnaire sample marked with the invalid label. The classifier can include one or more of a Naive Bayes classifier, a shallow neural network, and a logistic regression classifier. The training of the classifier can make the classification result of the classifier on the questionnaire sample in the training set tend to be consistent with the result label carried by the questionnaire sample, so as to enable the classifier and even the questionnaire sample cleaning model to have high accuracy in predicting the invalid probability of the questionnaire sample.

[0132] Figure 12 The step flowchart of constructing the questionnaire sample cleaning model according to the trained one or more classifiers in some embodiments of the present application is schematically shown. As shown in Figure 12 Based on the above embodiments, the step S1130 of constructing the questionnaire sample cleaning model according to the trained one or more classifiers can further include the following steps S1210 and S1220.

[0133] S1210. Obtain the classification accuracy, classification precision, and questionnaire sample recall rate of the trained one or more classifiers;

[0134] S1220. Select one or more classifiers to combine into the questionnaire sample cleaning model according to the classification accuracy, classification precision, and classification recall rate of the classifier.

[0135] The classification accuracy is the probability that the prediction result of the classifier is correct after the training set is used to train the classifier. The classification precision is the correct rate of the classifier in predicting a questionnaire sample as an invalid questionnaire sample after the training set is used to train the classifier. The classification recall rate is the ratio of the number of questionnaire samples that are actually invalid and are predicted as invalid questionnaire samples to the total number of questionnaire samples input into the classifier after the training set is used to train the classifier.

[0136] That is, the expression of the classification accuracy accuracy is:

[0137] accuracy = (TP + TN) / (TP + FN + FP + TN)

[0138] The expression of the classification precision precision is:

[0139] precision = TP / (TP + FP)

[0140] The expression of the recall rate recall is:

[0141] recall = TP / (TP + FN)

[0142] Wherein, TP is the number of samples that the classifier predicts as invalid questionnaire samples in the classification process. FN is the number of samples that the classifier does not predict as invalid questionnaire samples in the classification process. FP is the number of samples that the classifier predicts as invalid questionnaire samples in the classification process. TN is the number of samples that the classifier does not predict as invalid questionnaire samples in the classification process.

[0143] Then, according to the classification accuracy, classification precision and classification recall of the classifier, one or more classifiers are combined to form a questionnaire sample cleaning model. Specifically, the classifier with high classification accuracy and classification precision and not low recall is multiplied by a high weight, and the classifier with low classification accuracy and classification precision is multiplied by a low weight, and then the foregoing classifiers are combined to form a questionnaire sample cleaning model, which is beneficial to improve the prediction accuracy and precision of the questionnaire sample cleaning model, and enables the questionnaire sample cleaning model to have a high recall rate. Moreover, different feature data and invalid questionnaire samples can have different correlation coefficients. Before the foregoing classifiers are combined to form a questionnaire sample cleaning model, if the correlation coefficient of the feature data input into the classifier is high, the classifier can be multiplied by a high weight, and if the correlation coefficient of the feature data input into the classifier is low, the classifier can be multiplied by a low weight, thereby being beneficial to improve the prediction accuracy of the questionnaire sample cleaning model.

[0144] Figure 13 The application shows a schematic diagram of the cycle process of constructing a training set, model training and selection, model online application, and generating new data in some embodiments. Please refer to Figure 13 After the data is pulled for data cleaning and the training set is constructed, the user behavior data of the questionnaire samples in the training set can be aggregated and feature extracted at a preset granularity, that is, the feature data is obtained by feature engineering. Then, the model is trained using the feature data and the effect test is performed. The effect test includes detecting the classification accuracy, classification precision and classification recall of the questionnaire sample cleaning model and its classifier. After the effect test, the model is adjusted and stored. Then, the adjusted questionnaire sample cleaning model is tested by E2E (End-to-End) and is applied to the questionnaire sample cleaning model, which realizes the monitoring of the real-time recycled questionnaire pool, and the questionnaire owner can enjoy the automatic cleaning of invalid questionnaire samples without any operation.

[0145] Figure 14The flowchart of the step of obtaining the questionnaire samples marked with the result label and the user behavior data corresponding to the questionnaire samples in some embodiments of the present application to establish the training set is schematically shown. As shown in Figure 14 Based on the above embodiments, the step of obtaining the questionnaire samples marked with the result label and the user behavior data corresponding to the questionnaire samples in step S1110 to establish the training set can further include steps S1410-S1430.

[0146] S1410. Obtain the invalid questionnaire samples marked with the invalid label and add the invalid questionnaire samples to the training set;

[0147] S1420. Obtain the counter-label questionnaire samples marked with the valid label and add the counter-label questionnaire samples to the training set, wherein the counter-label questionnaire samples are the questionnaire samples misjudged as the invalid questionnaire samples by the questionnaire sample cleaning model;

[0148] S1430. Obtain the valid questionnaire samples marked with the valid label and add the valid questionnaire samples to the training set, so that the proportion of the invalid questionnaire samples in all the questionnaire samples with the result label in the training set is within the preset proportion range.

[0149] The invalid questionnaire samples marked with the invalid label can be directly marked by the questionnaire owner or marked by the professional research team during manual cleaning. The counter-label questionnaire samples marked with the valid label can be directly counter-labeled by the questionnaire owner. Specifically, the questionnaire owner can check the questionnaire samples determined as the invalid questionnaire samples by the questionnaire sample cleaning model and counter-label the actually valid questionnaire samples with the valid label to form the counter-label questionnaire samples. The valid questionnaire samples marked with the valid label can be randomly extracted from the database after removing the invalid questionnaire samples, or can be manually labeled by the questionnaire owner or marked by the professional research team during manual cleaning.

[0150] Please continue to refer to Figure 13 During the monitoring of the real-time questionnaire pool, new data can be generated, such as the questionnaire samples marked with the invalid label by the questionnaire owner, the questionnaire samples marked with the invalid label by the professional research team, the counter-label questionnaire samples formed by the questionnaire owner counter-labeling the questionnaire samples misjudged as the invalid questionnaire samples, and the valid questionnaire samples obtained by randomly sampling the valid questionnaire samples marked with the valid label after removing the invalid questionnaire samples.

[0151] The valid questionnaire samples marked with the valid label are obtained, and the valid questionnaire samples are added to the training set, so that the proportion of the invalid questionnaire samples in all questionnaire samples with result labels in the training set is in a preset proportion range, and the number of the valid questionnaire samples marked with the valid label and the number of the invalid questionnaire samples marked with the invalid label are in a balanced proportion range, which is beneficial to provide a high-quality questionnaire sample training set, optimize the training of the classifier and the questionnaire sample cleaning model, and thus improve the accuracy of the questionnaire sample processing method in the embodiment of the present application in determining questionnaire samples.

[0152] Figure 15 The step flowchart of determining the questionnaire sample as an invalid questionnaire sample when the invalid probability of the questionnaire sample is greater than the preset threshold in some embodiments of the present application is schematically shown. As shown in Figure 15 On the basis of the above embodiments, the invalid questionnaire samples include the high-probability invalid questionnaire samples and the suspected invalid questionnaire samples, and the step S240 of determining the questionnaire sample as an invalid questionnaire sample when the invalid probability of the questionnaire sample is greater than the preset threshold can further include the following steps S1510-S1520.

[0153] S1510. When the invalid probability of the questionnaire sample is greater than a first preset threshold and less than or equal to a second preset threshold, the questionnaire sample is determined as a suspected invalid questionnaire sample.

[0154] S1520. When the invalid probability of the questionnaire sample is greater than the second preset threshold, the questionnaire sample is determined as a high-probability invalid questionnaire sample.

[0155] Further, the invalid questionnaire samples can further include a first-level probability invalid questionnaire sample, a second-level probability invalid questionnaire sample, a third-level probability invalid questionnaire sample and a fourth-level probability. When the invalid probability of the questionnaire sample is less than or equal to the first preset threshold, the questionnaire sample is determined as the first-level probability invalid questionnaire sample; when the invalid probability of the questionnaire sample is greater than the first preset threshold and less than or equal to the second preset threshold, the questionnaire sample is determined as the second-level probability invalid questionnaire sample; when the invalid probability of the questionnaire sample is greater than the second preset threshold and less than or equal to a third preset threshold, the questionnaire sample is determined as the third-level probability invalid questionnaire sample; and when the invalid probability of the questionnaire sample is greater than the third preset threshold, the questionnaire sample is determined as the fourth-level probability invalid questionnaire sample. The first preset threshold is less than the second preset threshold, and the second preset threshold is less than the third preset threshold. The invalid probability of the questionnaire sample in the fourth-level probability invalid questionnaire sample is the largest, the invalid probability of the questionnaire sample in the third-level probability invalid questionnaire sample is the second, the invalid probability of the questionnaire sample in the second-level probability invalid questionnaire sample is the third, and the invalid probability of the questionnaire sample in the first-level probability invalid questionnaire sample is the smallest.

[0156] For example, the first preset threshold is 25%, the second preset threshold is 50%, and the third preset threshold is 75%. Then, the first level probability [0%, 25%], the second level probability (25%, 50%), the third level probability (50%, 75%), and the fourth level probability (75, 100%) of the invalid questionnaire samples can be visually displayed by using a line chart, a column chart, or a sector chart. Specifically, the first level probability invalid sample is a high-probability valid questionnaire sample, the second level probability invalid sample is a suspected valid questionnaire sample, the third level probability invalid sample is a suspected invalid questionnaire sample, and the fourth level probability invalid sample is a high-probability invalid questionnaire sample.

[0157] In some embodiments, the change in the invalid probability of the questionnaire samples collected in some preset specific time periods can also be analyzed, which is beneficial to the time monitoring and analysis of the questionnaire sample collection and invalidity by the questionnaire owner.

[0158] Figure 16 A part of the step flowchart of the questionnaire sample processing method in some embodiments of the present application is schematically shown. As shown in Figure 16 Based on the above embodiments, the questionnaire sample processing method can further include the following steps S1610-S1630.

[0159] S1610. Collecting questionnaire samples from one or more channels;

[0160] S1620. Obtaining the invalid probability of the questionnaire samples in each channel, respectively;

[0161] S1630. Diagnosing the channel quality of each channel according to the distribution of the invalid probability of the questionnaire samples corresponding to the channel.

[0162] Figure 17a A distribution diagram of the invalid probability of the questionnaire samples corresponding to a first channel in some embodiments of the present application is schematically shown. Please refer to Figure 17a The horizontal axis is the invalid probability of the questionnaire samples, and the vertical axis is the proportion of the sample quantity of the questionnaire samples with a certain invalid probability collected by the channel to the total sample quantity of the questionnaire samples collected by the channel. It can be seen that the wave peak of the waveform formed by the column chart is on the left. According to the distribution of the low invalid probability, it can be judged that the channel quality of the channel is high, the questionnaire samples collected by the channel have more valid samples, the data quality is good, and the possibility of the invalidity of the questionnaire samples collected by the channel is low.

[0163] Figure 17b A distribution diagram of the invalid probability of the questionnaire samples corresponding to a second channel in some embodiments of the present application is schematically shown. Please refer to Figure 17bThe horizontal axis is the invalid probability of the questionnaire sample, and the vertical axis is the proportion of the sample quantity of the questionnaire sample under a certain invalid probability of the channel to the total questionnaire sample collected by the channel. It can be seen that the peaks of the wave formed by the column chart are in the middle. According to the distribution of the medium invalid probability, it can be judged that the channel quality of the channel is medium, the questionnaire sample collected by the channel has a medium quantity of valid samples, and the questionnaire sample collected by the channel has a medium possibility of invalidity.

[0164] Figure 17c A distribution condition chart of the invalid probability of the questionnaire sample corresponding to the third channel in some embodiments of the present application is schematically shown. Please refer to Figure 17b The horizontal axis is the invalid probability of the questionnaire sample, and the vertical axis is the proportion of the sample quantity of the questionnaire sample under a certain invalid probability of the channel to the total questionnaire sample collected by the channel. It can be seen that the peaks of the wave formed by the column chart are in the middle. According to the distribution of the medium invalid probability, it can be judged that the channel quality of the channel is medium, the questionnaire sample collected by the channel has a medium quantity of valid samples, and the questionnaire sample collected by the channel has a medium possibility of invalidity.

[0165] According to the above, the channel quality of the first channel is the highest, the channel quality of the second channel is the second, and the channel quality of the third channel is the lowest.

[0166] Therefore, it can help the questionnaire owner to judge and select a questionnaire collection channel with high quality, and can improve the data reliability of the collected questionnaire sample. Moreover, the user research department often needs to purchase and use a sample library from an external channel, and the questionnaire sample processing method of some embodiments can efficiently help the user research team to find the quality problem of the channel sample library at a low cost, so as to facilitate the use of the sample library formed by the questionnaire collected by the questionnaire collection channel with high quality.

[0167] It should be noted that although the steps of the method in the present application are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired result. In addition or alternatively, some steps can be omitted, a plurality of steps can be combined into one step, and / or one step can be divided into a plurality of steps, etc.

[0168] The device embodiment of the present application is introduced below, which can be used to execute the questionnaire sample processing method in the above embodiments of the present application. Figure 18 A structural block diagram of the questionnaire sample processing device provided by the embodiments of the present application is schematically shown. As Figure 18As shown, the questionnaire sample processing device 1800 includes:

[0169] The sample data acquisition module 1810 is configured to acquire the questionnaire sample to be processed and the user behavior data corresponding to the questionnaire sample. The user behavior data includes the user's operation behavior during the process of completing the questionnaire sample and the timestamp of the operation behavior.

[0170] The feature data acquisition module 1820 is configured to aggregate and extract features from the user behavior data of the questionnaire sample at a preset granularity to obtain the feature data of the questionnaire sample.

[0171] The invalidity probability prediction module 1830 is configured to predict the invalidity probability of a questionnaire sample based on the feature data of the questionnaire sample.

[0172] The questionnaire sample determination module 1840 is configured to determine a questionnaire sample as invalid when the invalidity probability of the questionnaire sample is greater than a preset threshold.

[0173] In some embodiments of this application, based on the above embodiments, the feature data acquisition module includes:

[0174] The intermediate data acquisition unit is configured to aggregate and extract features from the user behavior data of the questionnaire sample at the question granularity to obtain intermediate data at the question granularity.

[0175] The feature data acquisition unit is configured to aggregate and extract features from intermediate data at the questionnaire sample granularity to obtain feature data of the questionnaire samples.

[0176] In some embodiments of this application, based on the above embodiments, the invalidity probability prediction module includes:

[0177] The feature data partitioning unit is configured to partition feature data into numerical feature data and non-numerical feature data;

[0178] The normalization processing unit is configured to normalize the numerical feature data to obtain the first preprocessed data;

[0179] The numerical processing unit is configured to perform numerical processing on non-numerical data to obtain second preprocessed data;

[0180] An invalidity probability prediction unit is configured to input first preprocessed data and second preprocessed data into a questionnaire sample cleaning model consisting of one or more classifiers to obtain the invalidity probability of the questionnaire sample, wherein the questionnaire sample cleaning model is used to predict the invalidity probability of the questionnaire sample.

[0181] In some embodiments of this application, based on the above embodiments, the questionnaire sample processing device further includes:

[0182] The training set establishing unit is configured to obtain questionnaire samples marked with result labels and user behavior data corresponding to the questionnaire samples to establish a training set, wherein the result labels include valid labels and invalid labels;

[0183] The classifier training unit is configured to train one or more classifiers using the training set;

[0184] The cleaning model constructing unit is configured to construct a questionnaire sample cleaning model according to the one or more trained classifiers.

[0185] In some embodiments of the present application, based on the above embodiments, the cleaning model constructing unit comprises:

[0186] The training effect obtaining sub-unit is configured to obtain classification accuracy, classification precision and questionnaire sample recall rate of the one or more trained classifiers;

[0187] The cleaning model combining sub-unit is configured to select one or more classifiers to combine into a questionnaire sample cleaning model according to the classification accuracy, the classification precision and the classification recall rate of the classifiers.

[0188] In some embodiments of the present application, based on the above embodiments, the training set establishing unit comprises:

[0189] The invalid questionnaire sample obtaining sub-unit is configured to obtain invalid questionnaire samples marked with invalid labels and add the invalid questionnaire samples to the training set;

[0190] The mislabeling questionnaire sample obtaining sub-unit is configured to obtain mislabeling questionnaire samples marked with valid labels and add the mislabeling questionnaire samples to the training set, the mislabeling questionnaire samples being questionnaire samples misjudged as invalid questionnaire samples by the questionnaire sample cleaning model;

[0191] The valid questionnaire sample obtaining sub-unit is configured to obtain valid questionnaire samples marked with valid labels and add the valid questionnaire samples to the training set, so that a proportion of the invalid questionnaire samples in all questionnaire samples with result labels in the training set is within a preset proportion range.

[0192] In some embodiments of the present application, based on the above embodiments, the sample data obtaining module comprises:

[0193] The data real-time grabbing unit is configured to grab real-time uploaded user behavior data, questionnaire samples and questionnaire topic data, wherein the user behavior data is bound with questionnaire labels and sample labels, the questionnaire samples are bound with questionnaire labels and sample labels, and the questionnaire topic data is bound with questionnaire labels and sample labels;

[0194] The data aggregation unit is configured to aggregate the user behavior data, the questionnaire sample, and the questionnaire question data bound with the same questionnaire label and the same sample label to obtain the questionnaire sample to be processed, the user behavior data corresponding to the questionnaire sample, and the question data corresponding to the questionnaire sample.

[0195] In some embodiments of the present application, based on the above embodiments, the invalid questionnaire sample includes a high-probability invalid questionnaire sample and a suspected invalid questionnaire sample; and the questionnaire sample determination module includes:

[0196] The suspected invalid determination unit is configured to determine the questionnaire sample as a suspected invalid questionnaire sample when the invalid probability of the questionnaire sample is greater than a first preset threshold and less than or equal to a second preset threshold.

[0197] The high-probability invalid determination unit is configured to determine the questionnaire sample as a high-probability invalid questionnaire sample when the invalid probability of the questionnaire sample is greater than the second preset threshold.

[0198] In some embodiments of the present application, based on the above embodiments, the questionnaire sample processing apparatus further includes:

[0199] The questionnaire sample recycling module is configured to recycle the questionnaire sample from one or more channels.

[0200] The invalid probability acquisition module is configured to acquire the invalid probability of the questionnaire sample in each channel respectively.

[0201] The channel quality diagnosis module is configured to diagnose the channel quality of each channel according to the distribution of the invalid probability of the questionnaire sample corresponding to each channel.

[0202] The specific details of the questionnaire sample processing apparatus provided in the embodiments of the present application have been described in detail in the corresponding method embodiments, which will not be repeated here.

[0203] It should be noted that the embodiments of the present application involve user behavior data and other user-related data. When the method of the present application is applied to specific products or technologies, the user's permission or consent is obtained, and the extraction, use, and processing of related data comply with local safety standards and local laws and regulations.

[0204] Figure 19 The computer system structure block diagram of the electronic device for implementing the embodiments of the present application is schematically shown.

[0205] It should be noted that, Figure 19 The computer system 1900 of the electronic device shown is only an example and should not limit the functions and use range of the embodiments of the present application.

[0206] As Figure 19As shown, the computer system 1900 includes a central processing unit 1901 (CPU), which can execute various appropriate actions and processes according to programs stored in a read-only memory 1902 (ROM) or loaded from a storage section 1908 into a random access memory 1903 (RAM). Various programs and data required for the operation of the system are also stored in the random access memory 1903. The central processing unit 1901, the read-only memory 1902, and the random access memory 1903 are connected to each other through a bus 1904. An input / output interface 1905 (I / O interface) is also connected to the bus 1904.

[0207] The following components are connected to the input / output interface 1905: an input section 1906 including a keyboard, a mouse, etc.; an output section 1907 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1908 including a hard disk, etc.; and a communication section 1909 including a network interface card such as a local area network card, a modem, etc. The communication section 1909 performs communication processing via a network such as the Internet. A drive 1910 is also connected to the input / output interface 1905 as necessary. A removable recording medium 1911 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 1910 as necessary, so that a computer program read therefrom is installed into the storage section 1908 as necessary.

[0208] In particular, according to embodiments of the present application, the processes described in the various method flowcharts can be implemented as a computer software program. For example, embodiments of the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication section 1909, and / or installed from the removable recording medium 1911. When the computer program is executed by the central processing unit 1901, various functions defined in the system of the present application are executed.

[0209] It should be noted that the computer-readable medium in the embodiments of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (Compact Disc Read-Only Memory, CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, device or apparatus. In this application, the computer-readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take on many forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium that can send, propagate, or transmit the program for use by or in connection with an instruction execution system, device or apparatus. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, or the like, or any suitable combination thereof.

[0210] The flowcharts and block diagrams in the drawings illustrate the possible implementation architectures, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment, or a portion of code that contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different orders than that shown in the figures. For example, two blocks noted in succession can actually be executed substantially concurrently, or they can sometimes be executed in reverse order, depending on the functionality involved. It should also be noted that each block in the flowcharts or block diagrams, and combinations of blocks in the flowcharts or block diagrams, can be implemented by special-purpose hardware-based systems that perform the specified functions or operations, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0211] It should be noted that, although several modules or units for a device for action execution are mentioned in the above detailed description, such a division is not mandatory. Indeed, according to an embodiment of the application, the features and functionalities of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functionalities of one module or unit described above can be further divided into several modules or units embodied.

[0212] Those skilled in the art can easily understand, through the above description of the embodiments, that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solutions according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.) or a network, and includes several instructions to make a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) execute the methods according to the embodiments of the present application.

[0213] Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the present application cover any and all variations of the present application that come within the scope of the present application, along with all equivalents thereof. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the application being indicated by the following claims.

[0214] It should be understood that the present application is not limited to the precise construction that has been described above and illustrated in the accompanying drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the present application. The scope of the present application is limited only by the appended claims.

Claims

1. A questionnaire sample processing method, characterized in that, The method includes: Obtain the questionnaire sample to be processed and the user behavior data corresponding to the questionnaire sample. The user behavior data includes the user's operation behavior during the process of completing the questionnaire sample and the timestamp of the time when the operation behavior occurs. In the user behavior data, each operation behavior is arranged in order from earliest to latest according to the corresponding timestamp. The user behavior data includes the question to which each operation behavior belongs. The user behavior data and questionnaire question data of the questionnaire sample are aggregated and feature extracted at the question granularity to obtain intermediate data at the question granularity; or, the user behavior data of the questionnaire sample are aggregated and feature extracted at the page-turning granularity to obtain intermediate data at the page-turning granularity. The intermediate data is aggregated and features are extracted at the questionnaire sample granularity to obtain the feature data of the questionnaire sample. The invalidity probability of the questionnaire sample is predicted based on the feature data of the questionnaire sample. The invalidity probability is predicted by a questionnaire sample cleaning model. The multiple classifiers in the questionnaire sample cleaning model are selected based on the classification accuracy, classification precision, and classification recall of the classifiers after training different types of classifiers with training samples. In the questionnaire sample cleaning model, the weight of each classifier is related to the classification accuracy, classification precision, and classification recall, and is positively correlated with the correlation coefficient corresponding to the feature data of the questionnaire sample input to the classifier. Different classifiers correspond to different types of feature data, and the correlation coefficient corresponding to the feature data represents the correlation coefficient between the feature data and the invalidity of the questionnaire sample. When the invalidity probability of the questionnaire sample is greater than a preset threshold, the questionnaire sample is determined to be an invalid questionnaire sample.

2. The questionnaire sample processing method according to claim 1, characterized in that, The prediction of the invalidity probability of the questionnaire sample based on the feature data includes: The feature data is divided into numerical feature data and non-numerical feature data; The numerical feature data is normalized to obtain the first preprocessed data; The non-numerical data is numericalized to obtain the second preprocessed data; The first preprocessed data and the second preprocessed data are input into a questionnaire sample cleaning model consisting of one or more classifiers to obtain the invalidity probability of the questionnaire sample, wherein the questionnaire sample cleaning model is used to predict the invalidity probability of the questionnaire sample.

3. The questionnaire sample processing method according to claim 2, characterized in that, Before inputting the first preprocessed data and the second preprocessed data into a questionnaire sample cleaning model composed of one or more classifiers to obtain the invalidity probability of the questionnaire samples, the method includes: Obtain questionnaire samples labeled with result tags and corresponding user behavior data to build a training set, wherein the result tags include valid tags and invalid tags; One or more classifiers are trained using the training set; A questionnaire sample cleaning model is constructed based on the trained one or more classifiers.

4. The questionnaire sample processing method according to claim 3, characterized in that, The step of constructing a questionnaire sample cleaning model based on the trained one or more classifiers includes: Obtain the classification accuracy, classification precision, and questionnaire sample recall of the one or more classifiers after training; Based on the classification accuracy, classification precision, and classification recall of the classifiers, one or more of the classifiers are selected to form the questionnaire sample cleaning model.

5. The questionnaire sample processing method according to claim 3, characterized in that, The step of obtaining questionnaire samples labeled with result tags and corresponding user behavior data to establish a training set includes: Obtain invalid questionnaire samples marked with invalid labels, and add the invalid questionnaire samples to the training set; Obtain back-labeled questionnaire samples with valid labels and add them to the training set. The back-labeled questionnaire samples are those questionnaire samples that were misjudged as invalid by the questionnaire sample cleaning model. Obtain valid questionnaire samples with valid labels and add them to the training set so that the proportion of invalid questionnaire samples to all questionnaire samples with result labels in the training set is within a preset range.

6. The questionnaire sample processing method according to claim 1, characterized in that, The process of obtaining the questionnaire samples to be processed and the corresponding user behavior data includes: Capture real-time uploaded user behavior data, questionnaire samples, and questionnaire question data, wherein the user behavior data is bound to questionnaire tags and sample tags, the questionnaire samples are bound to questionnaire tags and sample tags, and the questionnaire question data is bound to questionnaire tags and sample tags. The user behavior data, the questionnaire samples, and the questionnaire question data that are bound to the same questionnaire tags and the same sample tags are aggregated to obtain the questionnaire samples to be processed, the user behavior data corresponding to the questionnaire samples, and the question data corresponding to the questionnaire samples.

7. The questionnaire sample processing method according to claim 1, characterized in that, The invalid questionnaire samples include highly likely invalid questionnaire samples and suspected invalid questionnaire samples; The step of determining a questionnaire sample as invalid when the invalidity probability of the questionnaire sample is greater than a preset threshold includes: When the invalidity probability of the questionnaire sample is greater than the first preset threshold and less than or equal to the second preset threshold, the questionnaire sample is determined to be a suspected invalid questionnaire sample. When the invalidity probability of the questionnaire sample is greater than the second preset threshold, the questionnaire sample is determined to be a highly likely invalid questionnaire sample.

8. The questionnaire sample processing method according to claim 1, characterized in that, The method further includes: Retrieve questionnaire samples from one or more channels; Obtain the invalidity probability of questionnaire samples in each of the aforementioned channels; The channel quality is diagnosed based on the distribution of the invalidity probability of the questionnaire samples corresponding to each channel.

9. A questionnaire sample processing device, characterized in that, The device includes: The sample data acquisition module is configured to acquire questionnaire samples to be processed and user behavior data corresponding to the questionnaire samples. The user behavior data includes the user's operation behavior during the process of completing the questionnaire sample and the timestamp of the time when the operation behavior occurs. In the user behavior data, each operation behavior is arranged in order from earliest to latest according to the corresponding timestamp, and the user behavior data includes the question to which each operation behavior belongs. The feature data acquisition module is configured to aggregate and extract features from the user behavior data and questionnaire question data of the questionnaire sample at the question granularity to obtain intermediate data at the question granularity; or, to aggregate and extract features from the user behavior data of the questionnaire sample at the page-turning granularity to obtain intermediate data at the page-turning granularity; and to aggregate and extract features from the intermediate data at the questionnaire sample granularity to obtain the feature data of the questionnaire sample. The invalidity probability prediction module is configured to predict the invalidity probability of the questionnaire sample based on the feature data of the questionnaire sample. The invalidity probability is predicted by a questionnaire sample cleaning model. The questionnaire sample cleaning model contains multiple classifiers, which are selected based on the classification accuracy, classification precision, and classification recall of the classifiers after training different types of classifiers with training samples. In the questionnaire sample cleaning model, the weight of each classifier is related to the classification accuracy, classification precision, and classification recall, and is positively correlated with the correlation coefficient corresponding to the feature data of the questionnaire sample input to the classifier. Different classifiers correspond to different types of feature data, and the correlation coefficient corresponding to the feature data represents the correlation coefficient between the feature data and the invalidity of the questionnaire sample. The questionnaire sample determination module is configured to determine the questionnaire sample as invalid when the invalidity probability of the questionnaire sample is greater than a preset threshold.

10. A computer-readable medium, characterized in that, The system contains a computer program that, when executed by a processor, implements the questionnaire sample processing method as described in any one of claims 1-8.

11. An electronic device, characterized in that, include: processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the questionnaire sample processing method as described in any one of claims 1-8 by executing the executable instructions.

12. A computer program product, characterized in that, The system includes computer instructions that, when executed by the processor of a computer device, cause the computer device to perform the questionnaire sample processing method as described in any one of claims 1-8.

Images