Truth inference and online task allocation for numerical tasks in crowdsourcing scenarios

Abstract

The present disclosure provides a method for truth value inference and online task assignment for numerical tasks in a crowdsourcing scenario, including: constructing a potential topic for each numerical task in a numerical task set; initializing the topic-level reliability of the user for the potential topic of the sample numerical task according to the user's answer set for the sample numerical task and the true value of the sample numerical task; inferring the true value and topic distribution of each target numerical task according to the user's initial topic-level reliability and the user's answer set for the target numerical task; iteratively updating the user's topic-level reliability according to the true value and topic distribution of each target numerical task and the user's answer set for the target numerical task; and online assigning new numerical tasks according to the user's topic-level reliability obtained after iterative updating and the distribution matrix of the target numerical task. The present invention can accurately model the user reliability for numerical tasks in crowdsourcing scenarios and obtain high-quality crowdsourcing collection results.

Description

Technical Field

[0001] The present invention relates to the field of crowdsourcing technology, and in particular to a method and system for truth value inference and online task allocation for numerical tasks in a crowdsourcing scenario. Background Art

[0002] With the rapid development of sensor technology and wireless communication technology, various types of smart mobile terminals have gradually developed into mainstream computing and communication devices in people's daily lives. These rapidly popular mobile devices, combined with the increasingly improved mobile operating systems such as Android and iOS and the development of 3G / 4G / 5G communication technologies, have provided a foundation for the rise of crowdsourcing, a fast and low-cost data collection method. In crowdsourcing scenarios, the quality of user reliability modeling will have a significant impact on the results of crowdsourcing data collection. Designing corresponding truth inference and online task allocation algorithms based on accurate user reliability will make the data collected by crowdsourcing more accurate.

[0003] At present, there are a series of studies on user reliability in crowdsourcing scenarios, but the existing methods are not suitable for numerical tasks. The relevant technologies include user probability models, confusion matrix models and potential domain models. In classification tasks, existing works usually use user reliability models to evaluate the reliability of user answers, and design corresponding truth inference algorithms and online task allocation algorithms based on this reliability, but they cannot be directly used in numerical tasks:

[0004] 1) The existing reliability models are designed for classification tasks and are based on the accuracy of user answers. However, in numerical tasks, user reliability needs to be modeled based on the error of user answers. Therefore, the existing reliability models are not suitable for numerical tasks.

[0005] 2) Existing works do not take into account the existence of some potential clustering of numerical tasks and that users have different reliabilities in different task categories. Summary of the invention

[0006] Therefore, in the existing technology, there is no reliability modeling method suitable for numerical tasks in crowdsourcing scenarios, and the existing truth inference algorithms and online task allocation algorithms cannot be directly used in numerical tasks.

[0007] Therefore, there is a great need for a method for truth inference and online task allocation for numerical tasks in crowdsourcing scenarios, which can accurately model the user reliability of numerical tasks in crowdsourcing scenarios, and design an efficient truth inference algorithm and online task allocation algorithm based on this user reliability model to obtain high-quality crowdsourcing collection results. In a first aspect of the embodiments of the present invention, a method for truth inference and online task allocation for numerical tasks in a crowdsourcing scenario is provided, comprising: constructing a potential topic for each numerical task in a numerical task set, wherein the numerical task set includes sample numerical tasks with known answers and target numerical tasks to be answered;

[0008] Initialize the topic-level reliability of the user's potential topic for the sample numerical task according to the user's answer set for the sample numerical task and the true value of the sample numerical task to obtain the user's initialized topic-level reliability;

[0009] Inferring the true value and topic distribution of each of the above target numerical tasks based on the user's initialized topic-level reliability and the user's answer set to the above target numerical tasks;

[0010] Iteratively updating the topic-level reliability of the user according to the true value and topic distribution of each of the target numerical tasks and the answer set of the user to the target numerical tasks;

[0011] New numerical tasks are assigned online according to the user's topic-level reliability obtained after iterative updating and the distribution matrix of the above target numerical tasks.

[0012] In one embodiment of the present invention, the topic-level reliability of the potential topic of the sample numerical task for the user is initialized based on the user's answer set for the sample numerical task and the true value of the sample numerical task, and obtaining the initialized topic-level reliability of the user includes:

[0013] A probabilistic graph model is obtained by modeling the errors of the users' answers to the sample numerical tasks and the initialization topic-level reliability of the users through Gaussian distribution;

[0014] Based on the above probabilistic graphical model, the Gibbs-EM algorithm is used to calculate the membership of the sample numerical task to the above potential topics and optimize the prior parameters of the sample numerical task topics. At the same time, the user-initialized topic-level reliability is calculated.

[0015] In another embodiment of the present invention, the iterative updating of the subject-level reliability of the user includes:

[0016] Obtaining the user's answer error for the target numerical task through the inferred true value of the target numerical task and the user's answer set for the target numerical task;

[0017] By combining the above answer error with the probabilistic graph model, the prior parameters of the topic-level reliability of the current user are iteratively updated.

[0018] In yet another embodiment of the present invention, the above-mentioned inferring the true value and topic distribution of each of the above-mentioned target numerical tasks based on the above-mentioned user's initial topic-level reliability and the user's answer set to the above-mentioned target numerical task includes:

[0019] For the latent variable distribution of the true value and the subject distribution under the current parameters of the above target numerical task, calculate the expectation of the log-likelihood function of the maximum posterior probability;

[0020] The expected derivative of the above log-likelihood function yields the true value and topic distribution of the above target numerical task.

[0021] In yet another embodiment of the present invention, the online allocation of new numerical tasks based on the user's subject-level reliability obtained after iterative updating and the distribution matrix of the target numerical tasks comprises:

[0022] Discretize the continuous numerical answers to the above target numerical tasks according to the answer range and accuracy requirements of the above target numerical tasks provided by the task demander;

[0023] Based on the above discretized values ​​and in accordance with the above updated user subject-level reliability and the user's answer to the above current target numerical task, a distribution matrix of the current task is established, and at the same time, a true value distribution of the above target numerical task is calculated according to the above distribution matrix;

[0024] The uncertainty of the true value distribution of the target numerical task is measured by entropy, and the uncertainty when the target numerical task is assigned to the user is calculated based on the user's possible answers to the target numerical task;

[0025] According to the uncertainty of the true value distribution of the target numerical task and the uncertainty of the target numerical task when it is assigned to the user, a target numerical task with the largest difference between the true value distribution of the target numerical task and the uncertainty when it is assigned to the user is selected through a greedy algorithm and assigned to the user.

[0026] In yet another embodiment of the present invention, the discretization of the continuous numerical answer to the target numerical task comprises:

[0027]

[0028] Among them: t represents the task; e t，min represents the minimum value of the answer range of task t; δ t represents the accuracy requirement of task t; a i Represents the answer set A for task tt Any answer.

[0029] In another embodiment of the present invention, the distribution matrix of the current task is established based on the discretized values ​​and according to the updated user subject-level reliability and the user's answer to the current target value task, including:

[0030] For each of the above target numerical tasks, define a distribution matrix M t Store the distribution of topics and ground truth values ​​for the above target numerical task:

[0031]

[0032] Where: k represents the latent topic of the target numerical task, and the element m in the matrix t，k，l indicates that task t belongs to latent topic k and the true value is c t，l probability; represents the candidate answer after the discretization of the target numerical task t; B t represents the discretized answer set of the target numerical task t; Represents the subject-level reliability of users for numerical tasks.

[0033] In yet another embodiment of the present invention, calculating the true value distribution of the target numerical task according to the distribution matrix includes:

[0034] Regularize the above distribution matrix:

[0035]

[0036] Calculate the true value of the target numerical task based on the above distribution matrix:

[0037]

[0038] in:

[0039] In yet another embodiment of the present invention, the uncertainty of the distribution of the true value of the target numerical task is measured by entropy, including:

[0040]

[0041] In yet another embodiment of the present invention, according to the uncertainty of the true value distribution of the target numerical task and the uncertainty of the target numerical task when assigned to the user, selecting the target numerical task with the largest difference between the true value distribution of the target numerical task and the uncertainty when the target numerical task is assigned to the user through a greedy algorithm and assigning the target numerical task to the user includes:

[0042] The user specifies the maximum number of tasks that can be completed this time;

[0043] According to the user's possible answers to the above target numerical task, the uncertainty when the above target numerical task is assigned to the user is calculated by weighted average method:

[0044]

[0045] Among them, w represents the user;

[0046] Calculate the difference between the true value distribution of the above target numerical task and the uncertainty when the above target numerical task is assigned to the user:

[0047]

[0048] The target numerical tasks with the largest difference among the maximum number of tasks are selected through a greedy algorithm and assigned to the user.

[0049] In a second aspect of the embodiments of the present invention, a crowdsourcing system for truth value inference and online task assignment methods for numerical tasks in a crowdsourcing scenario is provided, comprising: a user reliability module, used to construct a potential topic for each numerical task in a numerical task set, wherein the numerical task set includes sample numerical tasks with known answers and target numerical tasks to be answered; initializing the topic-level reliability of the potential topic of the sample numerical task by the user according to the answer set of the sample numerical task and the truth value of the sample numerical task, to obtain the initialized topic-level reliability of the user; a truth value inference module, used to infer the truth value and topic distribution of each target numerical task according to the initialized topic-level reliability of the user and the answer set of the user to the target numerical task; iteratively updating the topic-level reliability of the user according to the truth value and topic distribution of each target numerical task and the answer set of the user to the target numerical task; and an online task assignment module, online assigning new numerical tasks according to the topic-level reliability of the user obtained after iterative updating and the distribution matrix of the target numerical task.

[0050] The present invention provides a probabilistic graphical model of user reliability for numerical tasks in crowdsourcing scenarios, and based on the probabilistic graphical model, provides a corresponding truth value inference algorithm and an online task allocation algorithm, which greatly improves the quality of crowdsourcing task results.

[0051] The advantages of the method provided by the present invention are reflected in that, based on the phenomenon of potential clustering of numerical tasks, the user reliability is measured by using the answer error, and the user reliability probability graph model can be accurately characterized; at the same time, based on the obtained user reliability model probability graph model, a new truth value inference algorithm is provided, which improves the accuracy of inference; in addition, the uncertainty of the true value distribution of each task is measured by entropy, and an online task allocation algorithm is provided to minimize the uncertainty performance to effectively improve the allocation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a flowchart of a method for truth value inference and online task allocation for numerical tasks in a crowdsourcing scenario provided by an embodiment of the present invention.

[0053] Figure 2 It is a flowchart of a method for initializing a user's subject-level reliability of a potential subject for a sample numerical task provided by an embodiment of the present invention.

[0054] Figure 3 It is a flowchart of a method for iteratively updating user's subject-level reliability provided by an embodiment of the present invention.

[0055] Figure 4 It is a module diagram of a truth value inference and online task allocation system for numerical tasks in a crowdsourcing scenario provided by an embodiment of the present invention.

[0056] Figure 5 It is a schematic diagram of a computer system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0058] Combine the following Figure 1 A method for truth inference and online task allocation for numerical tasks in a crowdsourcing scenario according to an exemplary embodiment of the present invention is described.

[0059] In one embodiment of the present invention, Figure 1 As shown, the method for truth value inference and online task allocation for numerical tasks in a crowdsourcing scenario provided by an embodiment of the present invention includes operations S101 to S105.

[0060] In operation S101 , a potential topic of each numerical task in a numerical task set is constructed, wherein the numerical task set includes sample numerical tasks with known answers and target numerical tasks to be answered.

[0061] According to an embodiment of the present invention, a potential topic refers to a clustering relationship between tasks with the same truth value or the same text description, for example, including but not limited to a task of collecting Wi-Fi signal strength having a potential topic, that is, different device types or different frequency bands all belong to the clustering relationship of collecting Wi-Fi signal strength. Numerical tasks refer to problem tasks with numerical answers, for example, including but not limited to the temperature at a certain moment in the future.

[0062] In operation S102, the topic-level reliability of the user's potential topic for the sample numerical task is initialized according to the user's answer set for the sample numerical task and the true value of the sample numerical task to obtain the user's initialized topic-level reliability.

[0063] In operation S103 , the true value and topic distribution of each target numerical task are inferred according to the user's initial topic-level reliability and the user's answer set to the target numerical task.

[0064] According to an embodiment of the present invention, the true value of a numerical task is the correct answer to the current numerical task; the topic distribution is used to describe the topic and the degree of membership of each numerical task to the topic.

[0065] In operation S104, the topic-level reliability of the user is iteratively updated according to the true value and topic distribution of each target numerical task and the answer set of the user to the target numerical task.

[0066] In operation S105 , new numerical tasks are allocated online according to the user's subject-level reliability obtained after iterative updating and the distribution matrix of the target numerical tasks.

[0067] According to an embodiment of the present invention, a user reliability model for numerical tasks in crowdsourcing scenarios is provided, which can accurately model a user reliability probability graph model for numerical tasks in crowdsourcing scenarios, and based on the reliability model probability graph model, a corresponding truth value inference algorithm and an online task allocation algorithm are provided, which greatly improves the quality of crowdsourcing task results.

[0068] like Figure 2 As shown, in this embodiment, the topic-level reliability of the user's potential topic for the sample numerical task is initialized according to the user's answer set for the sample numerical task and the true value of the sample numerical task, and obtaining the user's initialized topic-level reliability includes operations S201 to S202.

[0069] In operation S201, assuming that there are K topics among the target numerical tasks, and a user shows consistent reliability on a set of tasks belonging to the same topic, the topics and the membership of each numerical task to the topic are described by Dirichlet distribution and multinomial distribution; the errors of the user's answers to the sample numerical tasks and the user's initialized topic-level reliability are modeled by Gaussian distribution to obtain a probabilistic graph model.

[0070] In operation S202, based on the probabilistic graphical model, the membership of the sample numerical task to each potential topic is calculated by the Gibbs-EM algorithm and the prior parameters of the sample numerical task topic are optimized, and the user-initialized topic-level reliability is calculated.

[0071] According to the embodiment of the present invention, the user reliability is measured by the potential clustering relationship existing in the numerical tasks and in combination with the error of the user's answer to the sample numerical tasks, so that the user's reliability probability graph model can be accurately characterized.

[0072] like Figure 3 As shown, in this embodiment, iteratively updating the subject-level reliability of a user includes operations S301 to S302.

[0073] In operation S301 , the user's answer error for the target numerical task is obtained through the inferred true value of the target numerical task and the user's answer set for the target numerical task.

[0074] In operation S302, the priori parameters of the topic-level reliability of the current user are iteratively updated by combining the answer error with the probabilistic graphical model.

[0075] In this embodiment, the true value and topic distribution of the target numerical task are calculated by the maximum expectation algorithm, including: for the latent variable distribution of the true value and topic distribution under the current parameters of the target numerical task, calculating the expectation of the log-likelihood function of the maximum posterior probability; and deriving the expectation of the log-likelihood function to obtain the true value and topic distribution of the target numerical task.

[0076] In this embodiment, the online allocation of new numerical tasks according to the user's subject-level reliability obtained after iterative updating and the distribution matrix of the target numerical tasks includes:

[0077] For each target numerical task, use [e t，min , e t，max ] to represent the answer range of the target numerical task, δ t Represents the accuracy requirement of the target numerical task t.

[0078] According to the answer range and accuracy requirements of the target numerical task provided by the task demander, the continuous numerical answer of the target numerical task is discretized, including:

[0079]

[0080] Among them: t represents the task; a i Represents the answer set A for task t t Any answer.

[0081] Based on the discretized values ​​and the updated user subject-level reliability and the user's answer to the current target numerical task, the distribution matrix of the current task is established. At the same time, the true value distribution of the target numerical task is calculated based on the distribution matrix. Specifically:

[0082] For each target numerical task, define a distribution matrix M t Store the distribution of topics and ground-truth values ​​for the target numerical task:

[0083]

[0084] Where: k represents the latent topic of the target numerical task, and the element m in the matrix t，k，l indicates that task t belongs to latent topic k and the true value is c t，l probability; represents the candidate answer after the discretization of the target numerical task t; B t represents the discretized answer set of the target numerical task t; Represents the subject-level reliability of users for numerical tasks.

[0085] Regularize the distribution matrix:

[0086]

[0087] Calculate the true value of the target numerical task based on the distribution matrix:

[0088]

[0089] in:

[0090] The uncertainty of the true value distribution of the target numerical task is measured by entropy, and the uncertainty when the target numerical task is assigned to the user is calculated based on the user's possible answers to the target numerical task, including:

[0091]

[0092] According to the uncertainty of the true value distribution of the target numerical task and the uncertainty when the target numerical task is assigned to the user, the target numerical task with the largest difference between the true value distribution of the target numerical task and the uncertainty when the target numerical task is assigned to the user is selected through a greedy algorithm and assigned to the user. Specifically:

[0093] The user specifies the maximum number of tasks that can be completed this time;

[0094] According to the user's possible answers to the target numerical task, the uncertainty when the target numerical task is assigned to the user is calculated by weighted average method:

[0095]

[0096] Among them, w represents the user;

[0097] Calculate the difference between the true value distribution of the target numerical task and the uncertainty when the target numerical task is assigned to the user:

[0098]

[0099] The target numerical tasks with the largest difference in the maximum number of tasks are selected through a greedy algorithm and assigned to the user.

[0100] According to an embodiment of the present invention, based on the user reliability probability graph model, a new truth value inference algorithm based on the EM (EXpectation-Maximization) algorithm is provided, which improves the accuracy of truth value inference; and, the uncertainty of the true value distribution of each task is measured by entropy, and an online task allocation algorithm is provided, which greatly improves the allocation efficiency.

[0101] After introducing the method of the exemplary embodiment of the present invention, next, refer to Figure 2 A truth inference and online task allocation system for numerical tasks in a crowdsourcing scenario according to an exemplary embodiment of the present invention is described.

[0102] like Figure 4 As shown, the truth value inference and online task allocation system for numerical tasks in a crowdsourcing scenario according to an embodiment of the present invention includes: a task demander 401, a user 402, an online task allocation module 403, a truth value inference module 404, and a user reliability module 405.

[0103] In one embodiment of the present invention, a crowdsourcing system for truth value inference and online task assignment methods for numerical tasks in a crowdsourcing scenario includes: a user reliability module 405, which is used to construct a potential topic for each numerical task in a numerical task set, wherein the numerical task set includes sample numerical tasks with known answers and target numerical tasks to be answered; based on the answer set of user 402 for the sample numerical tasks and the truth value of the sample numerical tasks, the topic-level reliability of user 402 for the potential topic of the sample numerical tasks is initialized to obtain the initialized topic-level reliability of user 402.

[0104] The truth value inference module 404 is used to infer the truth value and topic distribution of each target numerical task based on the initial topic-level reliability of the user 402 and the answer set of the user 402 to the target numerical task; according to the truth value and topic distribution of each target numerical task, and the answer set of the user 402 to the target numerical task, iteratively update the topic-level reliability of the user 402.

[0105] The online task assignment module 403 assigns new numerical tasks online according to the subject-level reliability of the user 402 obtained after iterative updating and the distribution matrix of the target numerical tasks.

[0106] In the method provided by the present invention, based on the phenomenon of potential clustering of numerical tasks, the reliability of user 402 is measured by using answer error, which can accurately characterize the reliability probability graph model of user 402; at the same time, based on the obtained reliability probability graph model of user 402, a new truth value inference algorithm is provided, which improves the accuracy of inference; in addition, the uncertainty of the true value distribution of each task is measured by entropy, and an online task allocation algorithm is provided to minimize the uncertainty performance, thereby effectively improving the allocation efficiency.

[0107] Figure 5 A block diagram of a computer system suitable for implementing the method described above according to an embodiment of the present disclosure is schematically shown. Figure 5 The computer system shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0108] like Figure 5 As shown, the computer system 500 according to the embodiment of the present disclosure includes a processor 501, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 502 or the program loaded from the storage part 508 to the random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include an onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to the embodiment of the present disclosure.

[0109] In RAM 503, various programs and data required for the operation of system 500 are stored. Processor 501, ROM 502 and RAM 503 are connected to each other through bus 504. Processor 501 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 502 and / or RAM 503. It should be noted that the program can also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0110] According to an embodiment of the present disclosure, the system 500 may further include an input / output (I / O) interface 505, which is also connected to the bus 504. The system 500 may further include one or more of the following components connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage section 508 as needed.

[0111] According to an embodiment of the present disclosure, the method flow according to an embodiment of the present disclosure can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the processor 501, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0112] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0113] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include, but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device.

[0114] For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 502 and / or the RAM 503 described above and / or one or more memories other than the ROM 502 and the RAM 503 .

[0115] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It can be understood by those skilled in the art that the features recorded in the various embodiments and / or claims of the present disclosure can be combined and / or combined in a variety of ways, even if such a combination or combination is not explicitly recorded in the present disclosure. In particular, without departing from the spirit and teaching of the present disclosure, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0116] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for truth inference and online task allocation for numerical tasks in a crowdsourcing scenario, characterized in that: include: Constructing a latent topic for each numerical task in a numerical task set, wherein the numerical task set includes sample numerical tasks with known answers and target numerical tasks to be answered; Initializing the subject-level reliability of the user's potential subject for the sample numerical task according to the user's answer set for the sample numerical task and the true value of the sample numerical task to obtain the user's initialized subject-level reliability; Inferring the true value and topic distribution of each target numerical task according to the user's initial topic-level reliability and the user's answer set to the target numerical task; Iteratively updating the topic-level reliability of the user according to the true value and topic distribution of each of the target numerical tasks and the answer set of the user to the target numerical tasks; New numerical tasks are allocated online according to the user's subject-level reliability obtained after iterative updating and the distribution matrix of the target numerical tasks.

2. The method according to claim 1, characterized in that Initializing the subject-level reliability of the user for the potential subject of the sample numerical task according to the user's answer set for the sample numerical task and the true value of the sample numerical task to obtain the user's initialized subject-level reliability includes: Modeling the error of the user's answer to the sample numerical task and the user's initialization subject-level reliability through Gaussian distribution to obtain a probabilistic graphical model; Based on the probabilistic graphical model, the membership of the sample numerical task to each potential topic is calculated by the Gibbs-EM algorithm and the prior parameters of the sample numerical task topic are optimized, and the user-initialized topic-level reliability is calculated; and / or The iterative updating of the subject-level reliability of the user comprises: Obtaining the user's answer error for the target numerical task through the inferred true value of the target numerical task and the user's answer set for the target numerical task; The prior parameters of the topic-level reliability of the current user are iteratively updated by combining the answer error with the probabilistic graph model.

3. The method according to claim 1, characterized in that The step of inferring the true value and topic distribution of each target numerical task according to the user's initial topic-level reliability and the user's answer set to the target numerical task includes: Calculate the expectation of the log-likelihood function of the maximum a posteriori probability for the latent variable distribution of the true value and the subject distribution under the current parameters of the target numerical task; The expected derivative of the log-likelihood function yields the true value and topic distribution of the target numerical task.

4. The method according to claim 1, characterized in that The online allocation of new numerical tasks according to the user's subject-level reliability obtained after iterative updating and the distribution matrix of the target numerical tasks includes: Discretizing the continuous numerical answer of the target numerical task according to the answer range and accuracy requirements of the target numerical task provided by the task demander; Establishing a distribution matrix of the current task based on the discretized numerical values ​​and in accordance with the updated user subject-level reliability and the user's answer to the current target numerical task, and calculating the true value distribution of the target numerical task based on the distribution matrix; Measuring the uncertainty of the true value distribution of the target numerical task by entropy, and calculating the uncertainty when the target numerical task is assigned to the user according to the user's possible answers to the target numerical task; According to the uncertainty of the true value distribution of the target numerical task and the uncertainty of the target numerical task when it is assigned to the user, a target numerical task with the largest difference between the true value distribution of the target numerical task and the uncertainty when the target numerical task is assigned to the user is selected through a greedy algorithm and assigned to the user.

5. The method according to claim 4, characterized in that Discretizing the continuous numerical answer to the target numerical task includes: Among them: t represents the task; e t，min represents the minimum value of the answer range of task t; δ t represents the accuracy requirement of task t; a i Represents the answer set A for task t t Any answer.

6. The method according to claim 4, characterized in that The step of establishing a distribution matrix of the current task based on the discretized numerical value and in accordance with the updated user subject-level reliability and the user's answer to the current target numerical value task includes: For each target numerical task, define a distribution matrix M t Store the distribution of topics and ground truth values ​​for the target numerical task: Where: k represents the latent topic of the target numerical task, and the element m in the matrix t，k，l indicates that task t belongs to latent topic k and the true value is c t，l probability; represents the candidate answer after the discretization of the target numerical task t; B t represents the discretized answer set of the target numerical task t; Represents the subject-level reliability of users for numerical tasks.

7. The method according to claim 4, characterized in that The calculating the true value distribution of the target numerical task according to the distribution matrix includes: Regularize the distribution matrix: Calculate the true value of the target numerical task based on the distribution matrix: in:

8. The method according to claim 4, characterized in that The step of measuring the uncertainty of the true value distribution of the target numerical task by entropy includes:

9. The method according to claim 4, characterized in that The step of selecting, according to the uncertainty of the true value distribution of the target numerical task and the uncertainty of the target numerical task when it is assigned to the user, a target numerical task having the largest difference between the true value distribution of the target numerical task and the uncertainty of the target numerical task when it is assigned to the user by a greedy algorithm and assigning the target numerical task to the user comprises: The user specifies the maximum number of tasks that can be completed this time; According to the user's possible answers to the target numerical task, the uncertainty when the target numerical task is assigned to the user is calculated by weighted average method: Among them, w represents the user; Calculate the difference between the true value distribution of the target numerical task and the uncertainty when the target numerical task is assigned to the user: The target numerical tasks with the largest number of tasks and the largest difference are selected through a greedy algorithm and assigned to the user.

10. A crowdsourcing system using the method according to any one of claims 1 to 9, characterized in that: include: A user reliability module is used to construct a potential topic for each numerical task in a numerical task set, wherein the numerical task set includes sample numerical tasks with known answers and target numerical tasks to be answered; based on the user's answer set for the sample numerical tasks and the true value of the sample numerical tasks, the topic-level reliability of the user's potential topic for the sample numerical tasks is initialized to obtain the user's initialized topic-level reliability; A truth value inference module, used to infer the truth value and topic distribution of each target numerical task according to the user's initial topic-level reliability and the user's answer set to the target numerical task; iteratively update the user's topic-level reliability according to the truth value and topic distribution of each target numerical task and the user's answer set to the target numerical task; The online task assignment module assigns new numerical tasks online according to the user's subject-level reliability obtained after iterative updating and the distribution matrix of the target numerical tasks.

Images