A method, device and electronic device for detecting cheating behavior of crowdsourcing users

By training a cheat model to detect the answer behavior of crowdsourcing users, the automated detection of cheating behavior of crowdsourcing users in online education is solved, time and cost savings are achieved, and the quality of question production is improved.

CN112598286BActive Publication Date: 2024-11-08BEIJING BAIGEFEICHI TECH LLC
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202011556332.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-12-23
Publication Date
2024-11-08
Estimated Expiration
2040-12-23

AI Technical Summary

Technical Problem

In existing online education, crowdsourcing users cheat in the process of answering subject questions, resulting in a large amount of manual monitoring, which consumes time and resources.

Method used

By obtaining effective historical answer data from crowdsourcing users, extracting user behavior information and question attribute information, training a cheat model to detect whether the current answer is cheating, realizing automated detection, and reducing the need for manual monitoring.

Benefits of technology

Effectively save monitoring time and labor costs, improve the production quality of questions, reduce cheating behavior, and improve the accuracy and effectiveness of cheating models.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112598286B_ABST
    Figure CN112598286B_ABST
Patent Text Reader

Abstract

The present invention belongs to the field of Internet technology, and provides a method, device and electronic device for detecting cheating behavior of crowdsourcing users, the method comprising: obtaining valid historical answering data of crowdsourcing users; using user behavior information and question attribute information corresponding to the valid historical answering data as features, and using whether the user corresponding to the valid historical answering data cheats as a label to train a cheating model; inputting the user behavior information and question attribute information of the current crowdsourcing user's answering into the trained cheating model to detect whether the current crowdsourcing user cheats in answering the question. Through the present invention, it is only necessary to extract the user behavior and question attributes of the crowdsourcing user in the process of answering questions to automatically detect whether the crowdsourcing user cheats, without manual monitoring, thereby effectively saving monitoring time and labor costs.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of Internet technology, and is particularly applicable to Internet online education technology. More specifically, it relates to a crowdsourcing user cheating behavior detection method, device, electronic device and computer-readable medium.

[0002] The crowdsourcing users referred to in the present invention refer to users who answer questions corresponding to each link of the production questions. For example, in the stage of judging duplicates, at least two questions will be given, and the crowdsourcing users will judge whether the given questions are the same; in the stage of dividing subjects, questions will be given, and the crowdsourcing users will judge the subject to which the question belongs; in the stage of dividing grades, questions will be given, and the crowdsourcing users will judge whether the question

[0003] The grade the project belongs to, etc. Background Art

[0004] As a supplement and enhancement to traditional school education, online education is becoming more and more accepted, with more and more students and teachers participating. In online education, in order to consolidate and test students' acceptance of classroom teaching knowledge, a large number of subject questions are often used as examples, exercises, variations, etc. This requires the production of a large number of subject questions to meet teaching needs.

[0005] The existing question production process can be divided into the following stages: judging duplicates, judging incompleteness, dividing subjects, dividing grades, answering, answer review, typesetting, typesetting review, splitting multiple questions, and splitting multiple questions for review. Each stage requires crowdsourcing users to answer different questions. For example, in the stage of judging duplicates, at least two questions will be given, and crowdsourcing users will judge whether the given questions are the same question; in the stage of judging incompleteness, the stem of the question will be given, and crowdsourcing users will judge whether the stem information is complete (whether the words or pictures are incomplete, etc.); in the stage of dividing subjects, questions will be given, and crowdsourcing users will judge the subject to which the question belongs; in the stage of dividing grades, questions will be given, and crowdsourcing users will judge the grade to which the question belongs, and so on. Crowdsourcing users usually receive questions and answer them through the client. Cheating is inevitable in the process of answering. If manual monitoring is carried out through cameras, it will consume a lot of manpower and time, resulting in a waste of resources. Summary of the invention

[0006] 1. Technical issues to be resolved

[0007] The present invention aims to solve the technical problem in the prior art that it is time-consuming and labor-intensive to manually monitor crowdsourcing users.

[0008] (II) Technical solution

[0009] In order to solve the above technical problems, one aspect of the present invention provides a method for detecting cheating behavior of crowdsourcing users, wherein the crowdsourcing users refer to users who answer questions corresponding to various links of production questions, and the method comprises the following steps:

[0010] Obtain valid historical answer data of crowdsourcing users;

[0011] The user behavior information and question attribute information corresponding to the valid historical answer data are used as features, and whether the user corresponding to the valid historical answer data cheats is used as a label to train a cheating model;

[0012] The user behavior information and question attribute information of the current crowdsourcing user in answering the question are input into the trained cheating model to detect whether the current crowdsourcing user is cheating in answering the question.

[0013] According to a preferred embodiment of the present invention, the valid historical answer data refers to question data with a correct answer data format and answer parameters within a preset normal range, and obtaining the valid historical answer data of the crowdsourcing user includes:

[0014] Collect historical answer data of crowdsourcing users;

[0015] Extract correct historical question answer data according to the format of historical question answer data;

[0016] Eliminate or modify data with abnormal answer parameters from the correct historical answer data according to the answer parameters of the historical questions and the answer parameter thresholds;

[0017] The correct historical answer data after eliminating or modifying the abnormal answer data will be regarded as the valid historical answer data.

[0018] According to a preferred embodiment of the present invention, the correct answers to historical questions are extracted online in real time.

[0019] Historical answer data;

[0020] Alternatively, offline asynchronous delay extracts correct historical answer data based on the correct answers to historical questions within a preset time period.

[0021] According to a preferred embodiment of the present invention, the answering parameters include: the number of answers given by the user on the day, the accuracy rate of the user on the day, the answering time given by the user on the day, and the user operations within the answering time on the day; correspondingly, the answering parameter thresholds include: the number of answers given by the user on the day threshold, the answering time threshold given by the user on the day, and the effective operations of the user, and the modifying of abnormal answering data from the correct historical answering data according to the answering parameters of the historical questions and the answering parameter thresholds includes:

[0022] When the number of answers given by a user on a given day exceeds the threshold for the number of answers given by the user on that day, the number of answers given by the user on that day will be modified to the average number of answers given by the user on that day, and the accuracy rate of the user on that day will be modified to the average accuracy rate of the user on that day.

[0023] When the user's answering time on the same day is greater than the user's answering time threshold on the same day, the user's answering time on the same day is modified to the user's average answering time;

[0024] User operations that meet the user's effective operations within the user's answering time on the day are extracted as the number of user operations.

[0025] According to a preferred embodiment of the present invention, the user behavior information includes at least one of: the number of answers given by the user on the day, the average number of answers given by the user on the day, the time the user answered on the day, the number of user operations, the accuracy rate of the user on the day, and the average accuracy rate of the user.

[0026] According to a preferred implementation manner of the present invention, the question attribute information includes: at least one of question difficulty, answering time for questions of the same type, answering time for the same question, subject, academic stage, and question completeness.

[0027] According to a preferred embodiment of the present invention, before using the user behavior information and question attribute information corresponding to the valid historical answer data as features and whether the user corresponding to the valid historical answer data cheats as a label to train the cheating model, the method further includes:

[0028] Configure the question difficulty scoring rules and question completeness scoring rules;

[0029] Obtaining question difficulty scoring data of the valid historical answer data according to the question difficulty scoring rule;

[0030] The question completeness scoring data of the valid historical answer data is obtained according to the question completeness scoring rule.

[0031] According to a preferred embodiment of the present invention, before inputting the user behavior information and question attribute information of the current crowdsourcing user in answering the question into the trained cheating model to detect whether the current crowdsourcing user cheated in answering the question, the method further includes:

[0032] The user behavior information and topic attribute information of the designated crowdsourcing user are collected as cross-validation samples; and the trained cheating model is validated by the cross-validation samples.

[0033] According to a preferred embodiment of the present invention, the method comprises:

[0034] Counting the number of cheating detections of each crowdsourcing user within a preset time period according to the detection results of the cheating model;

[0035] When the number of cheating detections is greater than the cheating threshold, the crowdsourcing user is banned.

[0036] According to a preferred embodiment of the present invention, the steps of producing questions include: any one of the steps of judging whether the questions are repeated or incomplete, judging whether the questions are divided into subjects or divided into study stages.

[0037] A second aspect of the present invention provides a crowdsourcing user cheating behavior detection device, wherein the crowdsourcing user refers to a user who answers questions corresponding to each link of a production question, and the device comprises: an acquisition module, for acquiring valid historical answer data of the crowdsourcing user;

[0038] A training module, used to use the user behavior information and question attribute information corresponding to the valid historical answer data as features, and whether the user corresponding to the valid historical answer data cheated as a label to train a cheating model;

[0039] The detection module is used to input the user behavior information and question attribute information of the current crowdsourcing user in answering the question into the trained cheating model to detect whether the current crowdsourcing user is cheating in answering the question.

[0040] A third aspect of the present invention provides an electronic device, comprising a processor and a memory, wherein the memory is used to store a computer executable program, and when the computer program is executed by the processor, the processor executes the described method.

[0041] A fourth aspect of the present invention further proposes a computer-readable medium storing a computer-executable program, wherein when the computer-executable program is executed, the method described is implemented.

[0042] (III) Beneficial effects

[0043] The present invention obtains the valid historical answering data of crowdsourcing users; then trains a cheating model based on the historical user behavior information, question attribute information, and label data of whether the historical user cheated extracted from the valid historical answering data, and detects whether the current crowdsourcing user cheated in answering questions through the trained cheating model. Through the present invention, it is only necessary to extract the user behavior and question attributes of the crowdsourcing user in the process of answering questions to automatically detect whether the crowdsourcing user cheats, without manual monitoring, thereby effectively saving monitoring time and labor costs.

[0044] The present invention extracts correct historical answer data according to the format of historical question answer data; and removes or modifies data with abnormal answer parameters from the correct historical answer data according to the answer parameters of the historical questions and the answer parameter threshold, thereby obtaining valid historical answer data, improving the effectiveness of cheating model training samples, and improving the accuracy of cheating model detection.

[0045] Accuracy.

[0046] The present invention counts the number of cheating detections of each crowdsourcing user within a preset time period according to the detection results of the cheating model; when the number of cheating detections is greater than the cheating threshold, the crowdsourcing user is banned. This can effectively reduce the cheating behavior of crowdsourcing users and improve the quality of question production. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a flow chart of a crowdsourcing user cheating behavior detection method of the present invention;

[0048] Figure 2 It is a structural schematic diagram of a crowdsourcing user cheating behavior detection device of the present invention;

[0049] Figure 3 is a schematic structural diagram of an electronic device according to an embodiment of the present invention;

[0050] Figure 4 is a schematic diagram of a computer-readable recording medium according to an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In the process of introducing specific embodiments, the detailed description of the structure, performance, effect or other characteristics is to enable those skilled in the art to fully understand the embodiments. However, it does not exclude that those skilled in the art can implement the present invention with a technical solution that does not contain the above-mentioned structure, performance, effect or other characteristics under certain circumstances.

[0052] The flowchart in the accompanying drawings is only an exemplary process demonstration, and does not mean that the solution of the present invention must include all the contents, operations and steps in the flowchart, nor does it mean that it must be executed in the order shown in the figure. For example, some operations / steps in the flowchart can be decomposed, some operations / steps can be combined or partially combined, etc. The execution order shown in the flowchart can be changed according to actual conditions without departing from the main purpose of the present invention.

[0053] Frames in the attached figure Figure 1 Generally, it refers to functional entities, which do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processing unit devices and / or microcontroller devices.

[0054] The same reference numerals in the drawings represent the same or similar elements, components or parts, and thus the repeated description of the same or similar elements, components or parts may be omitted below. It should also be understood that although the first, second, third and other attributives representing the numbers may be used herein to describe various devices, elements, components or parts, these devices, elements, components or parts should not be limited by these attributives. In other words, these attributives are only used to distinguish one from another. For example, the first device may also be called the second device, but it does not deviate from the essential technical solution of the present invention. In addition, the terms "and / or" and "and / or" refer to all combinations including any one or more of the listed items.

[0055] In order to solve the above technical problems, the present invention proposes a method for detecting cheating behavior of crowdsourcing users, wherein the crowdsourcing users refer to users who answer questions corresponding to each link of the production questions, and obtain the valid historical answering data of the crowdsourcing users; then train a cheating model based on the historical user behavior information, question attribute information, and label data of whether the historical users cheated extracted from the valid historical answering data, and use the trained cheating model to detect whether the current crowdsourcing user cheated in answering the questions. Through the present invention, it is only necessary to extract the user behavior and question attributes of the crowdsourcing user in the process of answering questions to automatically detect whether the crowdsourcing user cheated, without manual monitoring, thereby effectively saving monitoring time and labor costs.

[0056] The present invention extracts correct historical answer data according to the format of historical question answer data; and removes or modifies data with abnormal answer parameters from the correct historical answer data according to the answer parameters of the historical questions and the answer parameter thresholds, thereby obtaining valid historical answer data, improving the effectiveness of cheating model training samples, and improving the accuracy of cheating model detection.

[0057] In the present invention, the correct historical answer data can be extracted online in real time based on the correct answers to the historical questions; or the correct historical answer data can be extracted offline asynchronously and delayed based on the correct answers to the historical questions within a preset time period. Among them, the offline asynchronous delay processing can better count the historical answer data and realize batch processing.

[0058] In a specific method for modifying abnormal answer data, the answer parameters include: the number of answers given by the user on that day, the accuracy rate of the user on that day, the answer time given by the user on that day, and the user operations within the answer time given on that day; correspondingly, the answer parameter thresholds include: the threshold number of answers given by the user on that day, the threshold number of time given by the user on that day, and the effective operations of the user. When the number of answers given by the user on that day is greater than the threshold number of answers given by the user on that day, the number of answers given by the user on that day is modified to the historical average number of answers given by the user on that day, and the accuracy rate given by the user on that day is modified to the historical average daily accuracy rate given by the user; when the answer time given by the user on that day is greater than the threshold number of answer times given by the user on that day, the answer time given by the user on that day is modified to the average answer time given by the user; at the same time, the user operations that meet the effective operations of the user within the answer time given by the user on that day are extracted as the number of user operations.

[0059] In the present invention, the user behavior information includes at least one of the following: the number of answers given by the user on the day, the average number of answers given by the user, the time taken to answer the questions given on the day, the number of operations given by the user, the accuracy rate given by the user on the day, and the average accuracy rate given by the user. The question attribute information includes at least one of the following: the difficulty of the question, the time taken to answer the same type of question, the time taken to answer the same question, the subject, the stage of study, and the completeness of the question.

[0060] Before the cheating model training is performed, the present invention configures the question difficulty scoring rules and the question completeness scoring rules; obtains the question difficulty scoring data of the valid historical answer data according to the question difficulty scoring rules; and obtains the question completeness scoring data of the valid historical answer data according to the question completeness scoring rules, thereby realizing the extraction of historical question attribute information.

[0061] After the cheating model is trained, the present invention further collects user behavior information and question attribute information of the designated crowdsourcing user as a cross-validation sample; the trained cheating model is verified by the cross-validation sample to improve the accuracy of the cheating model. The present invention counts the number of cheating detections of each crowdsourcing user within a preset time period based on the detection results of the cheating model; when the number of cheating detections is greater than the cheating threshold, the crowdsourcing user is banned. It can effectively reduce the cheating behavior of crowdsourcing users and improve the quality of question production.

[0062] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0063] In the process of introducing specific embodiments, the detailed description of the structure, performance, effect or other characteristics is to enable those skilled in the art to fully understand the embodiments. However, it does not exclude that those skilled in the art can implement the present invention with a technical solution that does not contain the above-mentioned structure, performance, effect or other characteristics under certain circumstances.

[0064] The flowchart in the accompanying drawings is only an exemplary process demonstration, and does not mean that the solution of the present invention must include all the contents, operations and steps in the flowchart, nor does it mean that it must be executed in the order shown in the figure. For example, some operations / steps in the flowchart can be decomposed, some operations / steps can be combined or partially combined, etc. The execution order shown in the flowchart can be changed according to actual conditions without departing from the main purpose of the present invention.

[0065] Frames in the attached figure Figure 1 Generally, it refers to functional entities, which do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processing unit devices and / or microcontroller devices.

[0066] The same reference numerals in the drawings represent the same or similar elements, components or parts, and thus the repeated description of the same or similar elements, components or parts may be omitted below. It should also be understood that although the first, second, third and other attributives representing the numbers may be used herein to describe various devices, elements, components or parts, these devices, elements, components or parts should not be limited by these attributives. In other words, these attributives are only used to distinguish one from another. For example, the first device may also be called the second device, but it does not deviate from the essential technical solution of the present invention. In addition, the terms "and / or" and "and / or" refer to all combinations including any one or more of the listed items.

[0067] Figure 1 It is a flow chart of a method for detecting cheating behavior of crowdsourcing users of the present invention, wherein the crowdsourcing users refer to users who answer questions corresponding to each link of the production questions, such as Figure 1 As shown, the method comprises the following steps:

[0068] S1. Obtain the valid historical answer data of crowdsourcing users;

[0069] The valid historical answer data refers to the question data with the correct answer data format and the answer parameters within the preset normal range. The present invention eliminates errors and abnormal data by preprocessing the question data, thereby ensuring the effectiveness of the cheating model training samples.

[0070] Specifically, this step includes:

[0071] S11, collecting historical answer data of crowdsourcing users;

[0072] The historical answer data includes answers to historical questions, which can be obtained by collecting answers to questions given by historical crowdsourcing users in various stages of question production within a preset time period (e.g., three months). The stages of question production include: any one of the stages of judging the weight, judging the disability, dividing subjects, and dividing grades.

[0073] S12, extracting correct historical question answer data according to the format of historical question answer data;

[0074] Among them, the format of the history question answer data refers to the format requirements for the answers to history questions. Different production links of the questions have different corresponding answer format requirements. For example, the corresponding answer formats for the questions in the weighted and incomplete judgment links are: yes or no; the answer formats for the questions in the subject links are preset subject names, and the preset subject names can be: mathematics, English, Chinese and other subject names pre-stored in the system; the answer formats for the questions in the academic stage links are preset grades. Therefore, this step can first obtain the production link of the history questions, determine the format requirements of the answers to the questions according to the production link, and then determine whether the answers to the history questions meet the format requirements of the answers to the questions, thereby eliminating the history question data with incorrect answer formats.

[0075] In the present invention, the correct historical answer data can be extracted online in real time based on the correct answers to the historical questions; or the correct historical answer data can be extracted offline asynchronously and delayed based on the correct answers to the historical questions within a preset time period. Among them, the offline asynchronous delay processing can better count the historical answer data and realize batch processing.

[0076] S13, removing or modifying data with abnormal answer parameters from the correct historical answer data according to the answer parameters of the historical questions and the answer parameter thresholds;

[0077] In the present invention, the answering parameters are used to reflect the daily answering situation of the crowdsourcing users, and whether the answering data is abnormal is judged according to the daily answering situation of the crowdsourcing users. Exemplarily, the answering parameters include: the number of answers given by the user on the day, the accuracy rate of the user on the day, the answering time of the user on the day, and the user operations within the answering time on the day; correspondingly, the answering parameter thresholds include: the threshold of the number of answers given by the user on the day, the threshold of the answering time given by the user on the day, and the effective operation of the user. In the present invention, the user operation refers to any interactive behavior of the user in the process of answering questions, including but not limited to: click, drag, slide, preset gestures (such as rotating fingers at a predetermined angle) and other interactive behaviors; the effective operation of the user refers to the interactive behavior of the user with the terminal device components in the process of answering questions, including clicking components, dragging components, sliding components, and using preset gestures (such as rotating fingers at a predetermined angle) with components. The terminal components can be: a mouse, a display screen, a touch screen, etc.

[0078] Among them, the user answering parameters can first obtain the crowdsourcing user ID (such as the login account of the crowdsourcing user, etc.) who answered the correct historical questions, the answering timestamp of the crowdsourcing user ID, and the user operation within the answering timestamp; count the user's daily answer quantity and the user's daily answering time through the crowdsourcing user ID and the answering timestamp of the crowdsourcing user ID; determine the user's operation within the answering time of the day through the user operation within the answering timestamp; determine the user's daily correct rate by comparing the user's question answer with the correct answer. The answering parameter threshold is the answering parameter range of the crowdsourcing user's normal answering, which can be set according to the actual situation. For example, the user's daily answer quantity threshold can be set to 1,000 and the user's daily answering time threshold can be set to 8 hours.

[0079] Specifically, in a specific method of modifying abnormal answer data, if the user's daily answer volume is greater than the user's daily answer volume threshold, the user's daily answer volume is modified to the user's historical average daily answer volume, and the user's daily accuracy rate is modified to the user's historical average daily accuracy rate; when the user's daily answer time is greater than the user's daily answer time threshold, the user's daily answer time is modified to the user's average answer time; and user operations that meet the user's effective operations within the user's daily answer time are extracted as the number of user operations.

[0080] S14. The correct historical answer data after eliminating or modifying the abnormal answer data is used as the valid historical answer data.

[0081] S2, using the user behavior information and question attribute information corresponding to the valid historical answer data as features, and whether the user corresponding to the valid historical answer data cheated as a label to train a cheating model;

[0082] The user behavior information includes at least one of the number of answers given by the user on the day, the average number of answers given by the user on the day, the time the user answered on the day, the correct rate of the user on the day, and the average correct rate of the user. The number of answers given by the user on the day, the time the user answered on the day, and the number of user operations can first obtain the crowdsourcing user ID (such as the login account of the crowdsourcing user, etc.) who answered the correct historical questions, the answering timestamp of the crowdsourcing user ID, and the user operations within the answering timestamp; the number of answers given by the user on the day and the answering time of the user on the day are counted by the crowdsourcing user ID and the answering timestamp of the crowdsourcing user ID; the number of user operations is determined by the number of user operations within the answering timestamp; the correct rate of the user on the day is determined by comparing the answer to the question of the user on the day with the correct answer. The average number of answers given by the user is the average number of answers given by the crowdsourcing user on the day, and the average correct rate of the user is the average daily correct rate of the crowdsourcing user. The question attribute information includes at least one of the difficulty of the question, the answering time for the same type of questions, the answering time for the same question, the subject, the stage of study, and the completeness of the question. Among them, the answering time for the same type of questions can be determined based on the average answering time for the same type of questions by crowdsourcing users, and the answering time for the same question can be determined based on the average answering time for the same question by crowdsourcing users; the subject and grade can be obtained based on the attributes of the question. The difficulty of the question and the completeness of the question can be determined by configuring the question difficulty scoring rules and the question completeness scoring rules; the question difficulty scoring data of the valid historical answer data is obtained according to the question difficulty scoring rules; the question completeness scoring data of the valid historical answer data is obtained according to the question completeness scoring rules. For example, the question difficulty scoring rules can be to calculate the question difficulty score based on the question type, the question subject, the number of knowledge points involved in the question and the corresponding weights, and the question completeness scoring rules can be to calculate the question completeness score based on the completeness of the question structure, the completeness of the known conditions in the question and the corresponding weights.

[0083] After the cheating model is trained, the present invention also collects user behavior information and topic attribute information of designated crowdsourcing users as cross-validation samples; and verifies the trained cheating model through the cross-validation samples to improve the accuracy of the cheating model.

[0084] S3. Input the user behavior information and question attribute information of the current crowdsourcing user into the trained cheating model to detect whether the current crowdsourcing user cheats in answering the question.

[0085] Furthermore, the present invention can also count the number of cheating detections of each crowdsourcing user within a preset time period according to the detection results of the cheating model; when the number of cheating detections is greater than the cheating threshold, the crowdsourcing user is banned. For crowdsourcing users whose number of cheating detections is greater than 0 and less than the cheating threshold, their cheating answering behavior is regarded as invalid behavior, and their cheating answering data is used as historical answering data to optimize the cheating model.

[0086] Figure 2 It is a structural diagram of a crowdsourcing user cheating behavior detection device of the present invention, wherein the crowdsourcing user refers to a user who answers the corresponding questions of each link of the production question, such as Figure 2 As shown, the device comprises:

[0087] An acquisition module 21 is used to acquire valid historical answer data of crowdsourcing users;

[0088] A training module 22, for using the user behavior information and question attribute information corresponding to the valid historical answer data as features, and whether the user corresponding to the valid historical answer data cheats as a label to train a cheating model;

[0089] The detection module 23 is used to input the user behavior information and question attribute information of the current crowdsourcing user in answering the question into the trained cheating model to detect whether the current crowdsourcing user cheats in answering the question.

[0090] In a specific implementation, the valid historical answer data refers to the question data whose answer data format is correct and whose answer parameters are within a preset normal range. The acquisition module 21 includes:

[0091] The collection module is used to collect the historical answer data of crowdsourcing users;

[0092] An extraction module is used to extract correct historical question answer data according to the format of historical question answer data;

[0093] A removal and modification module is used to remove or modify data with abnormal answer parameters from the correct historical answer data according to the answer parameters of the historical questions and the answer parameter threshold;

[0094] The determination module is used to take the correct historical answer data after eliminating or modifying the abnormal answer data as the valid historical answer data.

[0095] The extraction module 22 extracts the correct historical answer data online in real time according to the correct answers to the historical questions; or, the extraction module 22 extracts the correct historical answer data offline in an asynchronous and delayed manner according to the correct answers to the historical questions within a preset time period.

[0096] In a specific implementation, the answering parameters include: the number of answers given by the user on that day, the accuracy rate of the user on that day, the answering time given by the user on that day, and the user operations within the answering time on that day; correspondingly, the answering parameter thresholds include: the threshold number of answers given by the user on that day, the threshold number of answering time given by the user on that day, and the effective operations of the user, and the elimination and modification module is specifically used for: when the number of answers given by the user on that day is greater than the threshold number of answers given by the user on that day, modifying the number of answers given by the user on that day to the user's historical average daily number of answers, and modifying the user's accuracy rate given on that day to the user's historical average daily accuracy rate; when the user's answering time on that day is greater than the threshold number of answering time given by the user on that day, modifying the user's answering time on that day to the user's average answering time; and extracting the user operations that meet the effective operations of the user within the user's answering time on that day as the number of user operations.

[0097] In the present invention, the user behavior information includes at least one of the following: the number of answers given by the user on the day, the average number of answers given by the user, the time taken to answer the questions given on the day, the number of operations given by the user, the accuracy rate given by the user on the day, and the average accuracy rate given by the user. The question attribute information includes at least one of the following: the difficulty of the question, the time taken to answer the same type of question, the time taken to answer the same question, the subject, the stage of study, and the completeness of the question.

[0098] Furthermore, the device also includes:

[0099] Configuration module, used to configure the scoring rules for question difficulty and question completeness;

[0100] A first acquisition module is used to acquire the question difficulty scoring data of the valid historical answer data according to the question difficulty scoring rule;

[0101] The second acquisition module is used to obtain the question completeness scoring data of the valid historical answer data according to the question completeness scoring rule.

[0102] The sub-collection module 24 is used to collect the user behavior information and topic attribute information of the designated crowdsourcing user as a cross-validation sample; and verify the trained cheating model through the cross-validation sample.

[0103] A statistics module 25, used to count the number of cheating detections of each crowdsourcing user within a preset time period according to the detection results of the cheating model;

[0104] The blocking module 26 is used to block the crowdsourcing user when the number of cheating detections is greater than a cheating threshold.

[0105] In the present invention, the steps of producing questions include: any one of the steps of judging whether the questions are repeated or incomplete, judging whether the questions are divided into subjects or divided into study stages.

[0106] Those skilled in the art will appreciate that the modules in the above device embodiments may be distributed in the device as described.

[0107] In the embodiment, corresponding changes can also be made and distributed in one or more devices different from the above embodiments.

[0108] Chunks can be combined into one module or further split into multiple submodules.

[0109] Figure 3 It is a structural schematic diagram of an electronic device of an embodiment of the present invention, the electronic device includes a processor and a memory, the memory is used to store a computer executable program, when the computer program is executed by the processor, the processor executes a crowdsourcing user cheating behavior detection method.

[0110] like Figure 3 As shown, the electronic device is presented in the form of a general computing device. The processor may be one or more and work in coordination. The present invention does not exclude distributed processing, that is, the processor may be dispersed in different physical devices. The electronic device of the present invention is not limited to a single entity, but may also be the sum of multiple physical devices.

[0111] The memory stores a computer executable program, which is usually a machine-readable code. The computer-readable program can be executed by the processor to enable the electronic device to perform the method of the present invention, or at least part of the steps in the method.

[0112] The memory includes a volatile memory, such as a random access memory unit (RAM) and / or a cache memory unit, and may also be a non-volatile memory, such as a read-only memory unit (ROM).

[0113] Optionally, in this embodiment, the electronic device further includes an I / O interface, which is used for the electronic device to exchange data with external devices. The I / O interface can represent one or more of several types of bus structures, including a storage unit bus or a storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of a variety of bus structures. It should be understood that Figure 3 The electronic device shown is only an example of the present invention, and the electronic device of the present invention may also include elements or components not shown in the above examples. For example, some electronic devices also include display units such as display screens, and some electronic devices also include human-computer interaction elements such as buttons, keyboards, etc. As long as the electronic device can execute the computer-readable program in the memory to implement the method of the present invention or at least part of the steps of the method, it can be considered as an electronic device covered by the present invention.

[0114] Figure 4 Schematic diagram of a computer readable recording medium according to an embodiment of the present invention. Figure 4As shown, a computer executable program is stored in a computer-readable recording medium, and when the computer executable program is executed, the crowdsourcing user cheating behavior detection method described above in the present invention is implemented. The computer-readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, device, or device. The program code contained on the readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0115] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0116] Through the above description of the implementation mode, it is easy for those skilled in the art to understand that the present invention can be implemented by hardware capable of executing a specific computer program, such as the system of the present invention, and the electronic processing unit, server, client, mobile phone, control unit, processor, etc. contained in the system. The present invention can also be implemented by a vehicle comprising at least a part of the above system or components. The present invention can also be implemented by computer software that executes the method of the present invention, such as control software executed by a microprocessor, an electronic control unit, a client, a server, etc. of a live broadcast device. However, it should be noted that the computer software that executes the method of the present invention is not limited to being executed by one or a specific hardware entity, and it can also be implemented in a distributed manner by unspecified specific hardware. For computer software, the software product can be stored in a computer-readable storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.), or it can be distributed and stored on the network, as long as it enables the electronic device to execute the method according to the present invention.

[0117] The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the present invention is not inherently related to any specific computer, virtual device or electronic device, and various general devices can also implement the present invention. The above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A crowdsourcing user cheating behavior detection method, characterized in that: The method comprises the following steps: Acquire valid historical answer data of crowdsourcing users, wherein the valid historical answer data refers to question data with correct answer data format and answer parameters within a preset normal range, including: historical answer data obtained by collecting answers to questions answered by historical crowdsourcing users in any link of the various links of question production in a preset time period; extract correct historical answer data from the format of historical answer data according to different format requirements of answers to questions corresponding to different links of question production; remove or modify data with abnormal answer parameters from correct historical answer data according to the answer parameters of historical questions and answer parameter thresholds; and use the correct historical answer data after removing or modifying abnormal answer data as valid historical answer data; The user behavior information and question attribute information corresponding to the valid historical answer data are used as features, and whether the user corresponding to the valid historical answer data cheats is used as a label to train the cheating model, wherein the question attribute information includes at least one of question difficulty, answer time for questions of the same type, answer time for the same question, subject, grade, and question completeness; The user behavior information and question attribute information of the current crowdsourcing user in answering the question are input into the trained cheating model to detect whether the current crowdsourcing user is cheating in answering the question.

2. The crowdsourcing user cheating behavior detection method according to claim 1 is characterized in that: The answering parameters reflect the daily answering situation of the crowdsourcing user, including: the number of answers given by the user on the day, the correct rate of the user on the day, the answering time of the user on the day, and the user operations within the answering time on the day; wherein the answering parameters count the number of answers given by the user on the day and the answering time of the user on the day through the answering timestamp in the crowdsourcing user ID of the correct historical question data and the user operations within the answering timestamp, and determine the user operations within the answering time on the day, and compare the answers to the questions on the day with the correct answers to determine the correct rate of the user on the day; wherein the user operation refers to any interactive behavior of the user during the answering process; The corresponding answer parameter thresholds include: the user's daily answer quantity threshold, the user's daily answer time threshold, and the user's effective operation; the user's effective operation refers to the user's interaction behavior with the terminal device component during the answering process.

3. The crowdsourcing user cheating behavior detection method according to claim 1 or 2, characterized in that: Extracting the correct historical answer data also includes: Extract the correct historical answer data based on the correct answers to historical questions online in real time; or, Offline asynchronous delay extracts the correct historical answer data based on the correct answers to historical questions within a preset time period.

4. The crowdsourcing user cheating behavior detection method according to claim 2 is characterized in that: Modifying abnormal answer data from correct historical answer data based on historical question answer parameters and answer parameter thresholds includes: When the number of answers given by a user on a given day exceeds the threshold for the number of answers given by the user on that day, the number of answers given by the user on that day will be modified to the average number of answers given by the user on that day, and the accuracy rate of the user on that day will be modified to the average accuracy rate of the user on that day. When the user's answering time on the same day is greater than the user's answering time threshold on the same day, the user's answering time on the same day is modified to the user's average answering time; User operations that meet the user's effective operations within the user's answering time on the day are extracted as the number of user operations.

5. The crowdsourcing user cheating behavior detection method according to claim 4 is characterized in that: The user behavior information includes at least one of the number of answers given by the user on the day, the average number of answers given by the user on the day, the time the user answered on the day, the number of user operations, the accuracy rate of the user on the day, and the average accuracy rate of the user.

6. The crowdsourcing user cheating behavior detection method according to claim 1, characterized in that: Before using the user behavior information and question attribute information corresponding to the valid historical answer data as features and whether the user corresponding to the valid historical answer data cheats as a label to train the cheating model, the method further includes: Configure a question difficulty scoring rule and a question completeness scoring rule; obtain the question difficulty scoring data of the valid historical answer data according to the question difficulty scoring rule; obtain the question completeness scoring data of the valid historical answer data according to the question completeness scoring rule; Before inputting the user behavior information and question attribute information of the current crowdsourcing user in answering the question into the trained cheating model to detect whether the current crowdsourcing user cheats in answering the question, the method further includes: The user behavior information and topic attribute information of the designated crowdsourcing user are collected as cross-validation samples; and the trained cheating model is validated by the cross-validation samples.

7. The crowdsourcing user cheating behavior detection method according to any one of claims 1 to 6, characterized in that: The method comprises: Counting the number of cheating detections of each crowdsourcing user within a preset time period according to the detection results of the cheating model; When the number of cheating detections exceeds the cheating threshold, the crowdsourcing user is banned; The steps of producing questions include: any one of the steps of judging the weight, judging the disability, dividing the questions into subjects and dividing the questions into academic stages.

8. A device for detecting cheating behavior of crowdsourcing users, wherein the crowdsourcing users refer to users who answer questions corresponding to each link of the production questions, characterized in that: The device comprises: An acquisition module is used to acquire valid historical answer data of crowdsourcing users, wherein the valid historical answer data refers to question data with a correct answer data format and answer parameters within a preset normal range, including: collecting historical answer data obtained by answering questions answered by historical crowdsourcing users in any link of the various links of producing questions in a preset time period; extracting correct historical answer data from the format of historical answer data according to different format requirements of answers to questions corresponding to different links of producing questions; eliminating or modifying data with abnormal answer parameters from correct historical answer data according to answer parameters of historical questions and answer parameter thresholds; and using the correct historical answer data after eliminating or modifying abnormal answer data as valid historical answer data; A training module, for using the user behavior information and question attribute information corresponding to the valid historical answer data as features, and whether the user corresponding to the valid historical answer data cheats as a label to train a cheating model, wherein the question attribute information includes at least one of question difficulty, answer time for questions of the same type, answer time for the same question, subject, grade, and question completeness; The detection module is used to input the user behavior information and question attribute information of the current crowdsourcing user in answering the question into the trained cheating model to detect whether the current crowdsourcing user is cheating in answering the question.

9. An electronic device, comprising a processor and a memory, wherein the memory is used to store a computer executable program, wherein: When the computer executable program is executed by the processor, the processor performs the method according to any one of claims 1 to 7.

10. A computer-readable medium storing a computer-executable program, characterized in that: When the computer executable program is executed, the method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Cheating detection method and device

    CN110570217A

  • Cheating identification method and device, electronic equipment and readable storage medium

    CN110659954A

  • Dynamic incentive calculation method and system, equipment for crowdsourcing system and medium

    CN111639969A

  • Method and system for determining result for task executed in crowd-sourced environment

    US20200380410A1

  • Operating method and device for crowdsourcing platform, computer device and storage medium

    WO2019200736A1