Method, device, computer equipment and storage medium for verifying answers to crowdsourcing tasks
Through technical means such as natural language semantic recognition and similarity calculation, combined with word vector clustering and word segmentation processing, the problem of low answer verification efficiency of crowdsourcing tasks is solved, and more efficient and accurate answer verification is achieved.
Patent Information
- Application Number
- CN202010135251.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-03-02
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2040-03-02
AI Technical Summary
The answer verification efficiency of crowdsourcing tasks in the prior art is inefficient, and it is impossible to effectively determine the pros and cons of the answers provided by different execution objects, resulting in the accuracy and efficiency of answer verification.
Through natural language semantic recognition, similarity calculation and preset consistency verification methods, combined with word vector clustering and word segmentation processing, the credibility value of each response answer is determined, and the similarity between the simulated answer and the reference answer is verified through the simulated answer and the similarity between the reference answer and the simulated answer , and finally confirm the verified response.
It improves the efficiency and accuracy of answer verification of crowdsourcing tasks, can effectively identify and confirm the answers that have been verified, and improves the reliability of answer verification.
Smart Images

Figure CN111460811B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a method, device, computer equipment and storage medium for verifying answers to crowdsourcing tasks. Background Art
[0002] With the rapid development of network technology, some companies or organizations often distribute crowdsourcing tasks to Internet objects through the Internet in order to obtain more creative information or solve some cross-domain problems efficiently and conveniently. They solve these problems through crowdsourcing tasks. Crowdsourcing tasks refer to distributing internal work tasks to external execution objects to complete in order to shorten the task completion time.
[0003] Since different execution objects have different learning methods and implementation logics, the execution results of different execution objects for the same task are different. At the same time, since the execution objects have learning capabilities, it is impossible to directly determine which execution objects provide answers that are definitely better than other execution objects, and the answers cannot be verified. At present, in order to ensure accurate task answers, a crowdsourcing task is generally distributed to multiple execution objects, and then the answers of each execution object are obtained, and then the answers are verified through manual screening. This method makes the answer verification accuracy of crowdsourcing tasks low and the answer verification efficiency low. How to improve the answer verification efficiency of crowdsourcing tasks has become a difficult problem that needs to be solved urgently. Summary of the invention
[0004] The purpose of the embodiments of the present application is to propose a method for verifying answers to crowdsourcing tasks, so as to solve the problem of low efficiency of answer verification in the prior art method of manually verifying crowdsourcing tasks.
[0005] In order to solve the above technical problems, the present application embodiment provides a method for verifying answers to crowdsourcing tasks, including:
[0006] From all the response answers obtained by the client, obtain each response answer corresponding to the target task as an initial answer, wherein each of the response answers corresponds to a response object;
[0007] By means of natural language semantic recognition, semantic recognition is performed on each of the initial answers to obtain semantic recognition results of N initial answers, where N is the number of the response objects and N is a positive integer;
[0008] The semantic recognition results are combined in pairs, and each combination is taken as a group of results. The similarity values between the semantic recognition results in each group of results are counted by similarity calculation. If the obtained similarity value is greater than a preset similarity threshold, the two semantic recognition results in the group are taken as the same type of reference answers to obtain M types of reference answers, where M≤N, and N is a positive integer;
[0009] Determine the credibility value of each type of reference answer through a preset consistency verification method;
[0010] From the credibility values of all the reference answers, select the credibility value with the largest value as the maximum credibility value, and compare the maximum credibility value with the preset standard value to obtain a comparison result;
[0011] If the comparison result is that the maximum credibility value is greater than or equal to the preset standard value, the reference answer corresponding to the maximum credibility value is used as the target answer, and the response answer corresponding to the target answer is confirmed as the verified response answer.
[0012] Furthermore, the semantic recognition of each of the initial answers is performed by natural language semantic recognition, and the semantic recognition results of the N initial answers obtained include:
[0013] Performing word segmentation processing on the initial answer by a preset word segmentation method to obtain basic word segmentations contained in the initial answer;
[0014] Convert the basic word segmentation into word vectors, and cluster the word vectors using a clustering algorithm to obtain a cluster center corresponding to each word vector;
[0015] The preset semantics corresponding to the cluster center corresponding to each of the word vectors is obtained as the semantic recognition result of the initial answer.
[0016] Furthermore, the initial answer is segmented by a preset segmentation method to obtain basic segmentations contained in the initial answer, including:
[0017] Perform word segmentation analysis on the basic sentence to obtain K word segmentation sequences;
[0018] For each of the word segmentation sequences, the occurrence probability of each word segmentation sequence is calculated according to the word sequence data of the preset training corpus, and the occurrence probabilities of K word segmentation sequences are obtained;
[0019] From the occurrence probabilities of the K word segmentation sequences, the word segmentation sequence corresponding to the occurrence probability reaching a preset probability threshold is selected as the target word segmentation sequence, and each word in the target word segmentation sequence is used as the basic word segmentation included in the initial answer.
[0020] Furthermore, after selecting the maximum credibility value from the credibility values of all the reference answers as the maximum credibility value and comparing the maximum credibility value with a preset standard value to obtain a comparison result, the answer verification method for the crowdsourcing task further includes:
[0021] If the comparison result is that the maximum credibility value is less than the preset standard value, the target task is acquired, and the target task is input into the preset model to obtain a simulated answer through the preset model;
[0022] For each type of reference answer, the similarity values between the reference answer and the simulated answer are counted to obtain M similarity values, and the similarity value with the largest numerical value is selected from the M similarity values. The reference answer corresponding to the similarity value with the largest numerical value is used as the target answer, and the response answer corresponding to the target answer is confirmed as the verified response answer.
[0023] Furthermore, after selecting the maximum credibility value from the credibility values of all the reference answers as the maximum credibility value and comparing the maximum credibility value with a preset standard value to obtain a comparison result, the answer verification method for the crowdsourcing task further includes:
[0024] The response object includes a first object and a second object, the response answer of the first object is a first response answer, the response answer of the second object is a second response answer, the first object and the second object both have corresponding preset weights, and the preset weight of the first object is less than the preset weight of the second object.
[0025] If the comparison result is that the maximum credibility value is less than the preset standard value, obtaining a first preset weight corresponding to the first response answer and a second preset weight corresponding to the second response answer;
[0026] Determine the credibility weight of each type of the reference answer according to the reference answer category to which the first response answer and the second response answer belong, the first preset weight, and the second preset weight;
[0027] Determining a weighted credibility value of each type of reference answer according to the credibility weight of each type of reference answer and a preset weight verification method;
[0028] The weighted credibility value with the largest numerical value is selected as the target weighted credibility value, the reference answer corresponding to the target weighted credibility value is obtained as the target answer, and the response answer corresponding to the target answer is confirmed as the verified response answer.
[0029] Furthermore, after selecting the weighted credibility value with the largest value as the target weighted credibility value, taking the reference answer corresponding to the target weighted credibility value as the target answer, and confirming the response answer corresponding to the target answer as the verified response answer, the method further includes:
[0030] Obtaining historical responses of the first object and the second object;
[0031] Determine the proportion of verified answers in the historical answers of the first object to obtain the answer accuracy of the first object, and determine the proportion of verified answers in the historical answers of the second object to obtain the answer accuracy of the second object;
[0032] According to the response accuracy and the preset classification threshold, the preset weight of the first object and the preset weight of the second object are updated to obtain an updated preset weight of the first object and an updated preset weight of the second object.
[0033] In order to solve the above technical problems, a technical solution adopted by the present invention is: to provide a crowdsourcing task answer verification device, comprising:
[0034] An initial answer acquisition module is used to acquire each answer corresponding to the target task from all the answer answers acquired by the client as an initial answer, wherein each of the answer answers corresponds to a response object;
[0035] A semantic recognition result module, used to perform semantic recognition on each of the initial answers by natural language semantic recognition to obtain semantic recognition results of N initial answers, where N is the number of the answer objects and N is a positive integer;
[0036] A reference answer classification module is used to combine the semantic recognition results in pairs, and take each combination as a group of results, and use a similarity calculation method to count the similarity values between the semantic recognition results in each group of results. If the obtained similarity value is greater than a preset similarity threshold, the two semantic recognition results in the group are taken as the same type of reference answers, and M types of reference answers are obtained, where M≤N, and N is a positive integer;
[0037] A credibility value statistics module, used to determine the credibility value of each type of reference answer through a preset consistency verification method;
[0038] A credibility value comparison module is used to select a credibility value with the largest value from the credibility values of all the reference answers as a maximum credibility value, and compare the maximum credibility value with a preset standard value to obtain a comparison result;
[0039] A simulated answer acquisition module, used for acquiring the target task if the comparison result shows that the maximum credibility value is less than the preset standard value, and inputting the target task into a preset model to obtain a simulated answer through the preset model;
[0040] The response answer verification module is used to count the similarity values between the reference answer and the simulated answer for each type of reference answer, obtain M similarity values, and select the similarity value with the largest numerical value from the M similarity values, take the reference answer corresponding to the similarity value with the largest numerical value as the target answer, and confirm the response answer corresponding to the target answer as the verified response answer.
[0041] Furthermore, the semantic recognition result module includes:
[0042] A basic word segmentation acquisition unit, used to perform word segmentation processing on the initial answer by a preset word segmentation method to obtain the basic word segmentations contained in the initial answer;
[0043] A word vector acquisition unit, used to convert the basic word segmentation into a word vector, and cluster the word vectors through a clustering algorithm to obtain a cluster center corresponding to each word vector;
[0044] The semantic recognition result unit is used to obtain the preset semantics corresponding to the cluster center corresponding to each of the word vectors as the semantic recognition result of the initial answer.
[0045] To solve the above technical problems, a technical solution adopted by the present invention is: to provide a computer device, including one or more processors; a memory for storing one or more programs, so that the one or more processors can implement the answer verification solution of any one of the crowdsourcing tasks described above.
[0046] To solve the above technical problems, a technical solution adopted by the present invention is: a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the answer verification solution for any of the crowdsourcing tasks described above is implemented.
[0047] A method for verifying answers to a crowdsourcing task in the above scheme obtains the answer of each answer object corresponding to the target task from all the answer answers obtained from the client as the initial answer; and performs semantic recognition on each initial answer, and then combines the semantic recognition results in pairs, and takes each combination as a group of results, and uses a similarity calculation method to count the similarity values between the semantic recognition results in each group of results. If the obtained similarity value is greater than a preset similarity threshold, the two semantic recognition results in the group are taken as reference answers of the same type to obtain M types of reference answers; and through a preset consistency verification method, the credibility value of each type of reference answer is determined; then the credibility value with the largest value is selected from the credibility values of all the reference answers as the maximum credibility value, and the maximum credibility value is compared with the preset standard value to obtain a comparison result; if the comparison result is that the maximum credibility value is greater than or equal to the preset standard value, the reference answer corresponding to the maximum credibility value is taken as the target answer, and the response answer corresponding to the target answer is confirmed as the verified response answer. By determining the maximum credibility value and performing similarity value statistics to obtain the similarity value with the largest value, and finally confirming the verified answer, the answer verification efficiency of the crowdsourcing task can be effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0049] Figure 1 It is a schematic diagram of the application environment of the method for verifying the answer to a crowdsourcing task provided in an embodiment of the present application;
[0050] Figure 2 A flowchart of an implementation method of a crowdsourcing task answer verification method provided in an embodiment of the present application;
[0051] Figure 3 It is a flowchart for implementing step 2 in the method for verifying the answer to a crowdsourcing task provided in an embodiment of the present application;
[0052] Figure 4 It is a flowchart for implementing step S21 in the method for verifying the answer to a crowdsourcing task provided in an embodiment of the present application;
[0053] Figure 5 It is an implementation flow chart after step S5 in the method for verifying the answer to a crowdsourcing task provided in an embodiment of the present application;
[0054] Figure 6It is another implementation flow chart after step S5 in the answer verification method for a crowdsourcing task provided in an embodiment of the present application;
[0055] Figure 7 is a flowchart for implementing step S57 in the method for verifying the answer to a crowdsourcing task provided in an embodiment of the present application;
[0056] Figure 8 It is a schematic diagram of a device for verifying answers to crowdsourcing tasks provided in an embodiment of the present application;
[0057] Fig. 9 It is a schematic diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.
[0059] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0060] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.
[0061] The present invention is described in detail below with reference to the accompanying drawings and embodiments.
[0062] See also Figure 1 , the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.
[0063] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, search applications, instant messaging tools, etc.
[0064] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.
[0065] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .
[0066] It should be noted that the answer verification method for a crowdsourcing task provided in the embodiment of the present application is generally executed by a server, and correspondingly, the answer verification device for a crowdsourcing task is generally set in the server.
[0067] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.
[0068] See also Figure 2 , Figure 2 A specific implementation of a method for verifying answers to crowdsourcing tasks is shown.
[0069] It should be noted that if there are substantially the same results, the method of the present invention is not limited to Figure 2 The process sequence shown is limited to the following steps:
[0070] S1: From all the responses obtained from the client, obtain each response answer corresponding to the target task as an initial answer, where each response answer corresponds to a response object.
[0071] Specifically, after generating or obtaining the target task, the server pushes the target task to multiple objects. After the object responds to the target task, it obtains a response answer, and the response answer is fed back to the server through the client. The server receives the response answer sent by each client through the network transmission protocol and uses it as the initial answer.
[0072] On the server side, the information of each response object is pre-stored. After the response object gives feedback on the task and gets a response answer, a mapping relationship between the response answer and the response object is established. After the initial answer is obtained, each initial corresponding response object is obtained, that is, a list of all objects that participate in the target task and feed back the response answer to the server is obtained.
[0073] The response object in this embodiment can be a preset network model (machine learning model or neural network model), or a preset computing engine or big data platform or other individuals with autonomous learning and discrimination capabilities, which are not specifically limited here. For the same crowdsourcing task, different objects will give their own response answers due to their different structures and implementation logics.
[0074] It should be noted that the server will push multiple tasks at the same time, and may receive responses from different tasks in the same time period. In order to facilitate the subsequent screening of crowdsourced reference answers, the server sets a task identifier for each task in advance. The target task identifier is used to select the initial answer corresponding to the target task from all responses.
[0075] For example, in one specific implementation, all responses obtained from the client include three task identifiers, namely, Task A, Task B, and Task C. The task identifier of the current crowdsourcing task is Task A. Therefore, the response with the task identifier of Task A is selected as the initial answer.
[0076] The task identifier may be specifically represented by characters, numbers, or a combination of characters and numbers. For example, a task identifier is “TPSB5201906280236”.
[0077] The target task in this embodiment is a statement of objective facts, for example, recognizing and transcribing text on a given image.
[0078] In this embodiment, a method for verifying an answer to a crowdsourcing task is executed on an electronic device (e.g. Figure 1 The server shown in the figure) can be connected by wired connection or wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultrawideband) connection, and other wireless connection methods currently known or to be developed in the future.
[0079] S2: performs semantic recognition on each initial answer by means of natural language semantic recognition to obtain semantic recognition results of N initial answers, where N is the number of response objects and N is a positive integer.
[0080] Specifically, through the method of natural language semantic recognition, semantic recognition is performed on each initial answer to obtain a semantic recognition result of each initial answer.
[0081] Among them, natural language semantic recognition is mainly achieved through the use of NLP. NLP (Natural Language Processing) is also called natural language processing. Since understanding natural language requires extensive knowledge about the outside world and the ability to apply and operate this knowledge, natural language cognition is also regarded as an AI-complete problem. NLP tasks mainly refer to some tasks involving the semantic understanding or parsing of natural language. Common NLP tasks include but are not limited to: speech recognition, Chinese word segmentation, part-of-speech tagging, text classification, parsing, automatic summarization, question answering, and information extraction.
[0082] S3: Combine the semantic recognition results in pairs, and take each combination as a group of results. Use similarity calculation to count the similarity values between the semantic recognition results in each group of results. If the obtained similarity value is greater than the preset similarity threshold, the two semantic recognition results in the group are taken as the same type of reference answers, and M types of reference answers are obtained, where M≤N, and N is a positive integer.
[0083] Specifically, by combining the semantic recognition results in pairs and taking each combination as a group of results, similarity calculation is applied to obtain the similarity value between each group of semantic recognition results. By presetting the similarity threshold, if it is greater than the preset similarity threshold, any two semantic recognition results are classified as the preset similarity threshold.
[0084] Among them, the preset similarity threshold is a preset setting of the server, which is used to classify different semantic recognition results respectively. The more reasonable preset similarity thresholds are 0.9, 0.8, 0.7, 0.6, etc. The specific settings can be set according to actual conditions and are not limited here. Preferably, the preset similarity threshold in this embodiment is 0.8.
[0085] The similarity calculation methods include, but are not limited to, Minkowski distance, Manhattan distance, Euclidean distance, Chebyshev distance, Hamming distance, and Mahalanobis distance.
[0086] Preferably, the Euclidean distance is used to calculate the similarity value between the semantic recognition results in each group of results; the Euclidean distance calculation can quickly and efficiently calculate the similarity value between the semantic recognition results in each group of results.
[0087] S4: Determine the credibility value of each type of reference answer through a preset consistency verification method.
[0088] Among them, the credibility value refers to the reliability of this type of reference answer.
[0089] In one embodiment, the following formula is used to calculate the credibility value of each type of reference answer:
[0090] CR 1 +CR 2 +…+CR m =1
[0091] P={P 1 , P 2 , ..., P m )
[0092]
[0093] Where P = {P 1 , P 2 , ..., P m} is the reference answer set, P m is the number of answers contained in the m-th reference answer, CR m is the credibility value corresponding to the mth type of reference answer, m is the number of reference answer categories, and n is the number of initial answers.
[0094] S5: Select the maximum credibility value from the credibility values of all reference answers as the maximum credibility value, and compare the maximum credibility value with the preset standard value to obtain a comparison result.
[0095] Specifically, the credibility value with the largest value is selected from the credibility values of each category of reference answers for comparison with a preset standard value, and based on the comparison result, it is determined whether there is a reference answer that meets the requirements among all the classified reference answers.
[0096] Among them, the preset standard value is a preset setting of the server, which is used to evaluate whether the credibility value meets the required value. The more reasonable range is (0.5, 1). The specific setting can be set according to the actual situation and is not limited here. Preferably, the preset standard value in this embodiment is 0.6.
[0097] The comparison result includes that the maximum credibility value is less than the preset standard value and the maximum credibility value is greater than or equal to the preset standard value.
[0098] S6: If the comparison result is that the maximum credibility value is greater than or equal to the preset standard value, the reference answer corresponding to the maximum credibility value is used as the target answer, and the response answer corresponding to the target answer is confirmed as the verified response answer.
[0099] Specifically, when the maximum credibility value is greater than or equal to the preset standard value, it means that the reference answer corresponding to the maximum credibility value is a trustworthy response answer, which is used as the target answer, and the response answer corresponding to the target answer is confirmed as a verified response answer.
[0100] The answer verification result is sent to the terminal devices 101, 102, and 103 so that the response object can know the result of the answer verification.
[0101] In this embodiment, the answer of each answer object corresponding to the crowdsourcing task is obtained from all the answer answers obtained from the client as the initial answer; and semantic recognition is performed on each initial answer, and then the semantic recognition results are combined in pairs, and each combination is used as a group of results. The similarity value between the semantic recognition results in each group of results is counted by similarity calculation. If the obtained similarity value is greater than the preset similarity threshold, the two semantic recognition results in the group are taken as the same type of reference answers to obtain M types of reference answers; and the credibility value of each type of reference answer is determined by a preset consistency verification method; then the maximum credibility value is selected from the credibility values of all the reference answers of each type to obtain a comparison result; if the comparison result is that the maximum credibility value is greater than or equal to the preset standard value, the reference answer corresponding to the maximum credibility value is taken as the target answer, and the answer answer corresponding to the target answer is confirmed as the verified answer answer. By determining the maximum credibility value and performing similarity value statistics, the similarity value with the largest value is obtained, and finally the verified answer answer is confirmed, which can effectively improve the answer verification efficiency of the crowdsourcing task.
[0102] See also Figure 3 , Figure 3A specific implementation of step S2 is shown. In step S2, semantic recognition is performed on each initial answer by natural language semantic recognition to obtain semantic recognition results of N initial answers, where N is the number of answer objects and N is a positive integer. The specific implementation process is described in detail as follows:
[0103] S21: Segment the initial answer using a preset segmentation method to obtain basic segmentations contained in the initial answer.
[0104] S22: Convert the basic word segmentation into word vectors, and cluster the word vectors through a clustering algorithm to obtain the cluster center corresponding to each word vector.
[0105] S23: Obtain the preset semantics corresponding to the cluster center corresponding to each word vector as the semantic recognition result of the initial answer.
[0106] In this embodiment, by performing word segmentation on the initial answer and clustering the word vectors through a clustering algorithm, a semantic recognition result of the initial answer is obtained, which makes the semantic recognition of the initial answer more accurate and further improves the verification accuracy of the response answer to the crowdsourcing task.
[0107] See also Figure 4 , Figure 4 A specific implementation of step S21 is shown. In step S21, the initial answer is segmented by a preset segmentation method to obtain the specific implementation process of the basic segmentation contained in the initial answer, which is described in detail as follows:
[0108] S211: Perform word segmentation analysis on the basic sentence to obtain K word segmentation sequences.
[0109] S212: For each word segmentation sequence, the occurrence probability of each word segmentation sequence is calculated according to the word sequence data of the preset training corpus, and the occurrence probabilities of K word segmentation sequences are obtained.
[0110] S213: From the occurrence probabilities of the K word segmentation sequences, a word segmentation sequence corresponding to the occurrence probability reaching a preset probability threshold is selected as the target word segmentation sequence, and each word in the target word segmentation sequence is used as a basic word segmentation included in the initial answer.
[0111] In this embodiment, by performing word segmentation analysis through basic sentences and calculating the occurrence probability of each word segmentation sequence, the basic word segmentations contained in the initial answer can be accurately obtained, so that the initial answer can obtain an accurate semantic recognition result.
[0112] See also Figure 5 , Figure 5 A specific implementation method after step S5 is shown, and the specific implementation process is described in detail as follows:
[0113] S51: If the comparison result is that the maximum credibility value is less than the preset standard value, the target task is obtained, and the target task is input into the preset model to obtain a simulated answer through the preset model.
[0114] Specifically, each type of target task has a preset corresponding task model. The task model does not have the ability to learn autonomously, but can obtain a simulated answer with an accuracy rate that meets the requirements. When the maximum credibility value is less than the preset standard value, the target task is obtained and input into the preset model to obtain a simulated answer through the preset model.
[0115] In a specific embodiment, a crowdsourcing task is text recognition and transcription in an image, the preset model is an optical character recognition matching model, the credibility value is between 0.65 and 0.75, and the preset credibility value is 0.55. In step S50, when the calculated credibility value is lower than 0.55, the preset model is used to obtain a simulated answer.
[0116] S52: For each type of reference answer, the similarity values between the reference answer and the simulated answer are counted to obtain M similarity values, and the similarity value with the largest value is selected from the M similarity values. The reference answer corresponding to the similarity value with the largest value is used as the target answer, and the response answer corresponding to the target answer is confirmed as the verified response answer.
[0117] Specifically, a simulated answer is obtained through a preset model, and each reference answer is compared with the simulated answer by similarity calculation to obtain a similarity value between the reference answer and the simulated answer, and the similarity values are arranged in order from large to small, and the reference answer with the largest similarity value is selected as the target answer, and the response answer corresponding to the target answer is confirmed as the verified response answer.
[0118] The similarity calculation method has been shown in step S3 and will not be repeated here.
[0119] In a specific implementation, the Euclidean distance is used to obtain the Euclidean distance between each reference answer and the simulated answer, such as 0.9, 0.8, 0.5, 0.2 and 0.1. Since the Euclidean distance is used, the smaller the Euclidean distance, the higher the similarity value between the reference answer and the simulated answer. Therefore, in the above Euclidean distance, the reference answer corresponding to the Euclidean distance of 0.1 should be selected as the target answer, and the response answer corresponding to the target answer should be confirmed as the verified response answer.
[0120] In this embodiment, a simulated answer is obtained by using a preset model, and a verified response answer is determined, which can further improve the accuracy of answer verification for crowdsourcing tasks.
[0121] See also Figure 6 , Figure 6 A specific implementation method after step S6 is shown, and the specific implementation process is described in detail as follows:
[0122] S53: The response object includes a first object and a second object. The response answer of the first object is a first response answer, and the response answer of the second object is a second response answer. Both the first object and the second object have corresponding preset weights, and the preset weight of the first object is less than the preset weight of the second object.
[0123] Specifically, in the server, the response object includes a first object and a second object, wherein the response answer of the first object is a first response answer, and the response answer of the second object is a second response answer, and both the first object and the second object have corresponding preset weights, and the preset weight of the first object is less than the preset weight of the second object.
[0124] Among them, the preset weight is based on the previous answers of the respondent to the crowdsourcing task. The answer weight set by the server can be used to evaluate the credibility of the respondent's answer.
[0125] It should be understood that by assigning preset weights to the first object and the second object, when the calculated credibility value does not meet the requirements (is less than the preset standard value), the response answers corresponding to the first object and the second object can be weighted according to the preset weight, so that the credibility value distribution of the response answers is more reasonable, which is conducive to improving the accuracy of the credibility value.
[0126] S54: If the comparison result is that the maximum credibility value is less than the preset standard value, a first preset weight corresponding to the first response answer and a second preset weight corresponding to the second response answer are obtained.
[0127] Specifically, if the maximum credibility value is less than the preset standard value, the response answer cannot be verified. It is necessary to obtain the first response answer and the second response answer of the first response object and the second response object, and assign different weights to the first response answer and the second response answer, and determine the credibility of the response answer based on the assigned weights.
[0128] Among them, the preset weight is a preset setting of the server, and a more reasonable range is (0.1, 1). The specific setting can be set according to the actual situation and is not limited here. Preferably, in this embodiment, the first preset weight is 0.6 and the second preset weight is 0.8.
[0129] S55: Determine the credibility weight of each type of reference answer according to the reference answer category to which the first response answer and the second response answer belong, the first preset weight, and the second preset weight.
[0130] Specifically, the crowdsourcing task has different types of answer questions, and correspondingly, there are reference answer categories to which the answer should belong. According to the reference answer categories to which the first answer and the second answer belong, the first preset weight and the second preset weight of the first answer and the second answer are identified, and finally the credibility weight of each type of reference answer is determined.
[0131] Among them, the credibility weight is the credibility ratio assigned by the server to each type of reference answer.
[0132] S56: Determine the weighted credibility value of each type of reference answer based on the credibility weight of each type of reference answer and a preset weight verification method.
[0133] The weighted credibility value is the reliability value corresponding to the response answer.
[0134] In one embodiment, the weighted credibility value of each type of reference answer is calculated using the following calculation method:
[0135] Assume the first answer is P 1 , then the reference answer set is P = {P 1 , P 2 , ..., P m}, the corresponding weighted credibility value is CR = (CR 1 +0.5, CR 2 , ..., CR m}.
[0136] Among them, P m is the number of responses contained in the m-th type of reference answer, CR is the weighted credibility value corresponding to the m-th type of reference answer, m is the number of reference answer categories, and n is the number of initial answers.
[0137] S57: Select the weighted credibility value with the largest numerical value as the target weighted credibility value, obtain the reference answer corresponding to the target weighted credibility value as the target answer, and confirm the response answer corresponding to the target answer as the verified response answer.
[0138] Specifically, by obtaining the maximum weighted credibility value, that is, the reference answer corresponding to the weighted credibility value is the best answer to the crowdsourcing task, it is used as the answer to the crowdsourcing task and confirmed as the verified answer.
[0139] In this embodiment, by determining the credibility weight of each type of reference answer and calculating the weighted credibility value of each type of reference answer, the reliability of each type of reference answer is obtained, thereby improving the accuracy of answer verification for crowdsourcing tasks.
[0140] See also Figure 7 , Figure 7A specific implementation of step S57 is shown. In step S57, the weighted credibility value with the largest value is selected as the target weighted credibility value, the reference answer corresponding to the target weighted credibility value is obtained as the target answer, and the response answer corresponding to the target answer is confirmed as the verified response answer, as described in detail as follows:
[0141] S571: Obtain historical response answers to the first object and the second object.
[0142] Specifically, the server divides the objects of the crowdsourcing task into a first object and a second object; the first object and the second object refer to objects of different categories, rather than specifically referring to one or two objects therein; by obtaining the historical response answers of the first object and the second object to the crowdsourcing task, the response levels of the first object and the second object to the crowdsourcing task can be known.
[0143] S572: Determine the proportion of verified answers in the historical answers of the first object to obtain the answer accuracy of the first object, and determine the proportion of verified answers in the historical answers of the second object to obtain the answer accuracy of the second object.
[0144] Specifically, by determining the proportion of verified responses in the historical responses of the first and second objects, the accuracy of the responses of the first and second objects is obtained, and the response capabilities of the first and second objects to the crowdsourcing task are further learned.
[0145] S573: According to the answer accuracy and the preset classification threshold, the preset weight of the first object and the preset weight of the second object are updated to obtain an updated preset weight of the first object and an updated preset weight of the second object.
[0146] Specifically, the response accuracy of the first object and the second object is obtained. If some of the first objects have a higher accuracy and the accuracy exceeds the preset classification threshold, this part of the first objects is adjusted to the second object; if some of the second objects have a lower accuracy and the accuracy is lower than the preset classification threshold, this part of the second objects is adjusted to the first object.
[0147] Among them, the preset classification threshold is a preset setting of the server, and a more reasonable range is (0.6, 0.9). The specific setting can be set according to the actual situation and is not limited here. Preferably, the preset classification threshold in this embodiment is 0.6.
[0148] In this embodiment, by judging the response accuracy of the first object and the second object, and updating the preset weight of the first object and the preset weight of the second object, the preset weight of the first object and the preset weight of the second object are updated, so that the responding object pays more attention to the response answer to the crowdsourcing task, further improving the accuracy and efficiency of the crowdsourcing task.
[0149] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.
[0150] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).
[0151] Please refer to Figure 8 , as a response to the above Figure 2 The present application provides an embodiment of a device for verifying the answer to a crowdsourcing task. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.
[0152] like Figure 8 As shown, the answer verification device of a crowdsourcing task in this embodiment includes: an initial answer acquisition module 81, a semantic recognition result module 82, a reference answer classification module 83, a credibility value statistics module 84, a credibility value comparison module 85 and a response answer verification module 86. Among them:
[0153] The initial answer acquisition module 81 is used to acquire each answer corresponding to the target task from all the answer answers acquired by the client as the initial answer, wherein each answer answer corresponds to a answer object;
[0154] The semantic recognition result module 82 is used to perform semantic recognition on each initial answer by natural language semantic recognition to obtain semantic recognition results of N initial answers, where N is the number of answer objects and N is a positive integer;
[0155] The reference answer classification module 83 is used to combine the semantic recognition results in pairs, and take each combination as a group of results, and use the similarity calculation method to count the similarity values between the semantic recognition results in each group of results. If the obtained similarity value is greater than a preset similarity threshold, the two semantic recognition results in the group are taken as the same type of reference answers, and M types of reference answers are obtained, where M≤N, and N is a positive integer;
[0156] The credibility value statistics module 84 is used to determine the credibility value of each type of reference answer through a preset consistency verification method;
[0157] The credibility value comparison module 85 is used to select the credibility value with the largest value from the credibility values of all reference answers as the maximum credibility value, and compare the maximum credibility value with a preset standard value to obtain a comparison result;
[0158] The response answer verification module 86 is used to use the reference answer corresponding to the maximum credibility value as the target answer if the comparison result is that the maximum credibility value is greater than or equal to the preset standard value, and confirm the response answer corresponding to the target answer as the verified response answer.
[0159] Furthermore, the semantic recognition result module 82 includes:
[0160] The basic word segmentation acquisition unit is used to perform word segmentation processing on the initial answer through a preset word segmentation method to obtain the basic word segmentations contained in the initial answer.
[0161] The word vector acquisition unit is used to convert the basic word segmentation into a word vector, and cluster the word vectors through a clustering algorithm to obtain the cluster center corresponding to each word vector.
[0162] The semantic recognition result unit is used to obtain the preset semantics corresponding to the cluster center corresponding to each word vector as the semantic recognition result of the initial answer.
[0163] Furthermore, the basic word segmentation acquisition unit includes:
[0164] The word segmentation sequence acquisition subunit is used to perform word segmentation analysis on the basic sentence to obtain K word segmentation sequences.
[0165] The word segmentation sequence occurrence probability subunit is used to calculate the occurrence probability of each word segmentation sequence according to the word sequence data of the preset training corpus for each word segmentation sequence, and obtain the occurrence probabilities of K word segmentation sequences.
[0166] The target word segmentation sequence determination subunit is used to select the word segmentation sequence corresponding to the occurrence probability reaching a preset probability threshold from the occurrence probabilities of K word segmentation sequences as the target word segmentation sequence, and use each word in the target word segmentation sequence as the basic word included in the initial answer.
[0167] Furthermore, the answer verification device for the crowdsourcing task also includes:
[0168] A simulation answer acquisition module is used to acquire a target task if the comparison result shows that the maximum credibility value is less than a preset standard value, and input the target task into a preset model to obtain a simulation answer through the preset model;
[0169] The target answer selection module is used to count the similarity values between the reference answers and the simulated answers for each type of reference answers, obtain M similarity values, and select the similarity value with the largest value from the M similarity values, and use the reference answer corresponding to the similarity value with the largest value as the target answer, and confirm the response answer corresponding to the target answer as the verified response answer.
[0170] Furthermore, the answer verification device for the crowdsourcing task also includes:
[0171] A preset weight setting module is used for answering objects including a first object and a second object, the answer to the first object is a first answer answer, the answer to the second object is a second answer answer, the first object and the second object both have corresponding preset weights, and the preset weight of the first object is less than the preset weight of the second object;
[0172] A preset weight acquisition module, used for acquiring a first preset weight corresponding to the first response answer and a second preset weight corresponding to the second response answer if the comparison result shows that the maximum credibility value is less than the preset standard value;
[0173] A credibility weight determination module, used to determine the credibility weight of each type of reference answer according to the reference answer category to which the first response answer and the second response answer belong, the first preset weight and the second preset weight;
[0174] A weighted credibility value determination module is used to confirm the weighted credibility value of each type of reference answer according to the credibility weight of each type of reference answer and a preset weight verification method;
[0175] The weighted credibility value selection module is used to select the weighted credibility value with the largest numerical value as the target weighted credibility value, obtain the reference answer corresponding to the target weighted credibility value as the target answer, and confirm the response answer corresponding to the target answer as the verified response answer.
[0176] Furthermore, the weighted credibility value selection module includes:
[0177] A historical response answer acquisition unit, used to acquire historical response answers of the first object and the second object;
[0178] The answer accuracy verification unit is used to determine the proportion of the answer that has passed the verification in the historical answer of the first object, so as to obtain the answer accuracy of the first object, and determine the proportion of the answer that has passed the verification in the historical answer of the second object, so as to obtain the answer accuracy of the second object;
[0179] The object updating unit is used to update the preset weight of the first object and the preset weight of the second object according to the response accuracy and the preset classification threshold, and obtain the updated preset weight of the first object and the updated preset weight of the second object.
[0180] To solve the above technical problems, the present application also provides a computer device. Fig. 9 , Fig. 9 This is a basic structural block diagram of the computer device in this embodiment.
[0181] The computer device 9 includes a memory 91, a processor 92, and a network interface 93 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 9 having three components, a memory 91, a processor 92, and a network interface 93, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field-Programmable Gate Array, FPGA), a digital processor (Digital Signal Processor, DSP), an embedded device, etc.
[0182] Computer devices can be computing devices such as desktop computers, notebooks, PDAs, and cloud servers. Computer devices can interact with users through keyboards, mice, remote controls, touch pads, or voice control devices.
[0183] The memory 91 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 91 can be an internal storage unit of the computer device 9, such as a hard disk or memory of the computer device 9. In other embodiments, the memory 91 can also be an external storage device of the computer device 9, such as a plug-in hard disk equipped on the computer device 9, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 91 can also include both the internal storage unit of the computer device 9 and its external storage device. In this embodiment, the memory 91 is generally used to store the operating system and various application software installed on the computer device 9, such as the program code of the X method, etc. In addition, the memory 91 can also be used to temporarily store various data that have been output or are to be output.
[0184] In some embodiments, the processor 92 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor 92 is generally used to control the overall operation of the computer device 9. In this embodiment, the processor 92 is used to run the program code stored in the memory 91 or process data, such as running the program code of a sampling method for a crowdsourcing task.
[0185] The network interface 93 may include a wireless network interface or a wired network interface, and the network interface 93 is generally used to establish a communication connection between the computer device 9 and other electronic devices.
[0186] The present application also provides another implementation, namely, providing a computer-readable storage medium, which stores a sampling program, and the sampling program can be executed by at least one processor to enable the at least one processor to perform the steps of a sampling method for a crowdsourcing task as described above.
[0187] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods of each embodiment of the present application.
[0188] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.
Claims
1. A method for verifying answers to crowdsourcing tasks, It is characterized in that include: From all the response answers obtained by the client, obtain each response answer corresponding to the target task as an initial answer, wherein each of the response answers corresponds to a response object; By means of natural language semantic recognition, semantic recognition is performed on each of the initial answers to obtain semantic recognition results of N initial answers, where N is the number of the response objects and N is a positive integer; The semantic recognition results are combined in pairs, and each combination is taken as a group of results. The similarity values between the semantic recognition results in each group of results are counted by similarity calculation. If the obtained similarity value is greater than a preset similarity threshold, the two semantic recognition results in the group are taken as the same type of reference answers to obtain M types of reference answers, where M≤N, and N is a positive integer; Determine the credibility value of each type of reference answer through a preset consistency verification method; From the credibility values of all the reference answers, select the credibility value with the largest value as the maximum credibility value, and compare the maximum credibility value with the preset standard value to obtain a comparison result; If the comparison result is that the maximum credibility value is greater than or equal to the preset standard value, the reference answer corresponding to the maximum credibility value is used as the target answer, and the response answer corresponding to the target answer is confirmed as the verified response answer; After selecting the maximum credibility value from all the credibility values of the reference answers as the maximum credibility value and comparing the maximum credibility value with a preset standard value to obtain a comparison result, the answer verification method for the crowdsourcing task further includes: The answer object includes a first object and a second object, the answer to the first object is a first answer, the answer to the second object is a second answer, the first object and the second object both have corresponding preset weights, and the preset weight of the first object is less than the preset weight of the second object; If the comparison result is that the maximum credibility value is less than the preset standard value, obtaining a first preset weight corresponding to the first response answer and a second preset weight corresponding to the second response answer; Determine the credibility weight of each type of the reference answer according to the reference answer category to which the first response answer and the second response answer belong, the first preset weight, and the second preset weight; Determining a weighted credibility value of each type of reference answer according to the credibility weight of each type of reference answer and a preset weight verification method; The weighted credibility value with the largest numerical value is selected as the target weighted credibility value, the reference answer corresponding to the target weighted credibility value is obtained as the target answer, and the response answer corresponding to the target answer is confirmed as the verified response answer.
2. The method for verifying the answer to a crowdsourcing task according to claim 1, It is characterized in that The semantic recognition of each of the initial answers is performed by natural language semantic recognition to obtain the semantic recognition results of N initial answers, including: Performing word segmentation processing on the initial answer by a preset word segmentation method to obtain basic word segmentations contained in the initial answer; Convert the basic word segmentation into word vectors, and cluster the word vectors using a clustering algorithm to obtain a cluster center corresponding to each word vector; The preset semantics corresponding to the cluster center corresponding to each of the word vectors is obtained as the semantic recognition result of the initial answer.
3. The method for verifying the answer to a crowdsourcing task according to claim 2, It is characterized in that The initial answer is segmented by a preset segmentation method to obtain the basic segmentations contained in the initial answer, including: Perform word segmentation analysis on the basic sentence to obtain K word segmentation sequences; For each of the word segmentation sequences, the occurrence probability of each word segmentation sequence is calculated according to the word sequence data of the preset training corpus, and the occurrence probabilities of K word segmentation sequences are obtained; From the occurrence probabilities of the K word segmentation sequences, the word segmentation sequence corresponding to the occurrence probability reaching a preset probability threshold is selected as the target word segmentation sequence, and each word in the target word segmentation sequence is used as the basic word segmentation included in the initial answer.
4. The method for verifying the answer to a crowdsourcing task according to claim 1, It is characterized in that After selecting the maximum credibility value from all the credibility values of the reference answers as the maximum credibility value and comparing the maximum credibility value with a preset standard value to obtain a comparison result, the answer verification method for the crowdsourcing task further includes: If the comparison result is that the maximum credibility value is less than the preset standard value, the target task is acquired, and the target task is input into the preset model to obtain a simulated answer through the preset model; For each type of reference answer, the similarity values between the reference answer and the simulated answer are counted to obtain M similarity values, and the similarity value with the largest numerical value is selected from the M similarity values. The reference answer corresponding to the similarity value with the largest numerical value is used as the target answer, and the response answer corresponding to the target answer is confirmed as the verified response answer.
5. The method for verifying the answer to a crowdsourcing task according to claim 1, It is characterized in that After selecting the weighted credibility value with the largest value as the target weighted credibility value, taking the reference answer corresponding to the target weighted credibility value as the target answer, and confirming the response answer corresponding to the target answer as the verified response answer, the answer verification method for the crowdsourcing task further includes: Obtaining historical responses of the first object and the second object; Determine the proportion of verified answers in the historical answers of the first object to obtain the answer accuracy of the first object, and determine the proportion of verified answers in the historical answers of the second object to obtain the answer accuracy of the second object; According to the answer accuracy and a preset classification threshold, the preset weight of the first object and the preset weight of the second object are updated to obtain an updated preset weight of the first object and an updated preset weight of the second object.
6. A device for verifying answers to crowdsourcing tasks, It is characterized in that The answer verification device for a crowdsourcing task performs the steps of the answer verification method for a crowdsourcing task as claimed in any one of claims 1 to 5, and the answer verification device for a crowdsourcing task comprises: An initial answer acquisition module is used to acquire each answer corresponding to the target task from all the answer answers acquired by the client as an initial answer, wherein each of the answer answers corresponds to a response object; A semantic recognition result module, used to perform semantic recognition on each of the initial answers by natural language semantic recognition to obtain semantic recognition results of N initial answers, where N is the number of the answer objects and N is a positive integer; A reference answer classification module is used to combine the semantic recognition results in pairs, and take each combination as a group of results, and use a similarity calculation method to count the similarity values between the semantic recognition results in each group of results. If the obtained similarity value is greater than a preset similarity threshold, the two semantic recognition results in the group are taken as the same type of reference answers, and M types of reference answers are obtained, where M≤N, and N is a positive integer; A credibility value statistics module, used to determine the credibility value of each type of reference answer through a preset consistency verification method; A credibility value comparison module is used to select a credibility value with the largest value from the credibility values of all the reference answers as a maximum credibility value, and compare the maximum credibility value with a preset standard value to obtain a comparison result; The response answer verification module is used to use the reference answer corresponding to the maximum credibility value as the target answer if the comparison result is that the maximum credibility value is greater than or equal to the preset standard value, and confirm the response answer corresponding to the target answer as the verified response answer.
7. The answer verification device for a crowdsourcing task according to claim 6, It is characterized in that The semantic recognition result modules include: A basic word segmentation acquisition unit, used to perform word segmentation processing on the initial answer by a preset word segmentation method to obtain the basic word segmentations contained in the initial answer; A word vector acquisition unit, used to convert the basic word segmentation into a word vector, and cluster the word vectors through a clustering algorithm to obtain a cluster center corresponding to each word vector; The semantic recognition result unit is used to obtain the preset semantics corresponding to the cluster center corresponding to each of the word vectors as the semantic recognition result of the initial answer.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method for verifying an answer to a crowdsourcing task as claimed in any one of claims 1 to 5 are implemented.
9. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method for verifying the answer to a crowdsourcing task as claimed in any one of claims 1 to 5.
