Processing method and processing device for crowdsourcing annotation data, equipment, medium and product

Abstract

The invention provides a crowdsourcing annotation data processing method and device, equipment, a medium and a product, and belongs to the technical field of data processing. The method comprises the following steps: processing to-be-processed first crowdsourcing annotation data based on a preset tag fine tuning model to obtain second crowdsourcing annotation data; and for the current iteration state, determining the cluster number of the current iteration state based on the plurality of target labeling clusters of the previous iteration state and the second crowdsourcing labeling data, determining a plurality of initial labeling clusters of the current iteration state according to the cluster number of the current iteration state, and removing outlier first labels in the initial labeling clusters of the current iteration state, obtaining a plurality of target labeling clusters in a current iteration state; under the condition that a preset stop condition is met, cluster aggregation is carried out according to the multiple target labeling clusters in the last iteration state and the second crowdsourcing labeling data, and a crowdsourcing processing result is obtained. According to the embodiment of the invention, the accuracy of crowdsourcing annotation data can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method for processing crowdsourced annotation data, a processing device for crowdsourced annotation data, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the continuous development of deep learning, especially the emergence of large models, data quality issues have received more attention. Crowdsourcing annotation, as an important way to obtain data, has the characteristics of low cost and high speed. However, due to the inconsistency of cognitive level or annotation intention among annotators, the obtained annotation results often contain noise, and the accuracy may not meet the usage requirements. Therefore, how to denoise the annotation results of different annotators to obtain high-quality annotation results has become an urgent problem to be solved in the field of crowdsourcing annotation. Summary of the invention

[0003] The present application provides a method for processing crowdsourced annotation data, a device for processing crowdsourced annotation data, an electronic device, a computer-readable storage medium, and a computer program product, which can improve the accuracy of crowdsourced annotation data.

[0004] In a first aspect, the present application provides a method for processing crowdsourced annotation data, which includes: processing first crowdsourced annotation data to be processed based on a preset label fine-tuning model to obtain second crowdsourced annotation data, wherein the first crowdsourced annotation data includes first labels obtained by multiple annotation subjects annotating target objects, and the second crowdsourced annotation data includes fine-tuned second labels corresponding to each of the first labels; for a current iteration state, determining the number of clusters of the current iteration state based on multiple target annotation clusters of the previous iteration state and the second crowdsourced annotation data, determining multiple initial annotation clusters of the current iteration state according to the number of clusters of the current iteration state, and removing outlier first labels in the initial annotation clusters of the current iteration state to obtain multiple target annotation clusters of the current iteration state, wherein the initial annotation clusters of the current iteration state include at least one first annotation, and the multiple target annotation clusters of the first iteration state are determined according to the first crowdsourced annotation data and the second crowdsourced annotation data; when a preset stop condition is met, cluster aggregation is performed according to the multiple target annotation clusters of the last iteration state and the second crowdsourced annotation data to obtain a crowdsourcing processing result.

[0005] In a second aspect, the present application provides a processing device for crowdsourced annotation data, which includes: a fine-tuning module, which is used to process the first crowdsourced annotation data to be processed based on a preset label fine-tuning model to obtain second crowdsourced annotation data, wherein the first crowdsourced annotation data includes a first label obtained by multiple annotation subjects annotating a target object, and the second crowdsourced annotation data includes a fine-tuned second label corresponding to each of the first labels; a clustering module, which is used to determine the number of clusters of the current iteration state based on multiple target annotation clusters of the previous iteration state and the second crowdsourced annotation data for the current iteration state, determine multiple initial annotation clusters of the current iteration state according to the number of clusters of the current iteration state, and remove outlier first labels in the initial annotation clusters of the current iteration state to obtain multiple target annotation clusters of the current iteration state, wherein the initial annotation cluster of the current iteration state includes at least one first annotation, and the multiple target annotation clusters of the first iteration state are determined according to the first crowdsourced annotation data and the second crowdsourced annotation data; an aggregation module, which is used to cluster the multiple target annotation clusters of the last iteration state and the second crowdsourced annotation data to obtain a crowdsourcing processing result when a preset stop condition is met.

[0006] In a third aspect, the present application provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, and one or more of the computer programs are executed by the at least one processor so that the at least one processor can execute the above-mentioned method for processing crowdsourcing annotation data.

[0007] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor / processing core, implements the above-mentioned method for processing crowdsourced annotation data.

[0008] In the embodiment provided by the present application, the first crowd-sourced annotation data to be processed is processed based on a preset label fine-tuning model to obtain second crowd-sourced annotation data, wherein the first crowd-sourced annotation data includes first labels obtained by multiple annotation subjects annotating target objects, and the second crowd-sourced annotation data includes fine-tuned second labels corresponding to each first label; for the current iteration state, the number of clusters of the current iteration state is determined based on multiple target annotation clusters of the previous iteration state and the second crowd-sourced annotation data, multiple initial annotation clusters of the current iteration state are determined according to the number of clusters of the current iteration state, and outlier first labels in the initial annotation clusters of the current iteration state are eliminated to obtain multiple target annotation clusters of the current iteration state, wherein the initial annotation cluster of the current iteration state includes at least one first annotation, and the multiple target annotation clusters of the first iteration state are determined according to the first crowd-sourced annotation data and the second crowd-sourced annotation data; when the preset stop condition is met, cluster aggregation is performed according to the multiple target annotation clusters of the last iteration state and the second crowd-sourced annotation data to obtain a crowd-sourcing processing result. It can be seen that in the embodiment of the present application, in the process of determining the number of clusters, full use is made of the fine-tuned second label and the information of the target object itself. Compared with the method of directly taking the maximum number of annotations as the number of clusters in the related art, the number of clusters can be determined more accurately, which provides an important basis for subsequent data processing; and in the clustering process, the outlier first label will be eliminated, which improves the rationality of clustering; finally, by clustering the target annotation cluster and the second crowdsourcing annotation data, the accuracy of the label data is further improved, so that higher quality crowdsourcing processing results can be obtained.

[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are used to provide a further understanding of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. The above and other features and advantages will become more apparent to those skilled in the art by describing detailed example embodiments with reference to the accompanying drawings. In the accompanying drawings:

[0011] Figure 1 A flowchart of a method for processing crowdsourced annotation data provided in an embodiment of the present application.

[0012] Figure 2 A schematic diagram of the processing process of a method for processing crowdsourced annotation data provided in Example 1 of the present application.

[0013] Figure 3A schematic diagram of the processing process of a method for processing crowdsourced annotation data provided in Example 2 of the present application.

[0014] Figure 4 A flowchart of a method for processing crowdsourced annotation data provided in an embodiment of the present application.

[0015] Figure 5 A block diagram of a device for processing crowdsourced annotation data provided in an embodiment of the present application.

[0016] Figure 6 A block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to enable those skilled in the art to better understand the technical solution of the present application, the exemplary embodiments of the present application are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0018] In the absence of conflict, the embodiments of the present application and the features therein may be combined with each other.

[0019] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0020] The terms used herein are only used to describe specific embodiments and are not intended to limit the present application. As used herein, the singular forms "a" and "the" are also intended to include plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "including" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof is not excluded. "Connected" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect.

[0021] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this application, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined herein.

[0022] With the continuous development of deep learning, especially the emergence of large models, data quality issues have received more attention. Crowdsourcing annotation, as an important way to obtain data, has the characteristics of low cost and high speed. However, due to the inconsistency of cognitive level or annotation intention among annotators, there is often noise in the obtained labeled data.

[0023] In order to remove noise and improve the accuracy of annotation results, in related technologies, the ability of the annotator and the difficulty of the task are usually modeled (for example, the worker probability or confusion matrix can be used to describe the ability of the annotator, and the task difficulty can be described using domain division), and then iterative calculations are performed based on the distribution of the annotators to calculate the ability of the annotator and the difficulty of the task, and finally the task results are weighted fused or optimally selected. In addition, in some other related technologies, task characteristics can also be considered to estimate the reliability of the annotation, thereby improving the reliability of the annotation results to a certain extent.

[0024] However, the above methods are mainly aimed at the aggregation processing of simple tags, and are difficult to apply to the aggregation process of complex tags. Secondly, the tags used in the aggregation methods in the related art are from the tag set belonging to the annotator, which will result in the performance upper limit of these methods being actually limited by the ability upper limit of the annotator.

[0025] In summary, how to further process the annotation results collected by different annotators to obtain high-quality annotation results has become an urgent problem to be solved.

[0026] In view of this, embodiments of the present application provide a method for processing crowdsourced annotation data, a device for processing crowdsourced annotation data, an electronic device, a computer-readable storage medium, and a computer program product.

[0027] According to the method for processing crowdsourced annotation data of an embodiment of the present application, the first crowdsourced annotation data to be processed is processed based on a preset label fine-tuning model to obtain second crowdsourced annotation data, the first crowdsourced annotation data includes first labels obtained by multiple annotation subjects annotating target objects, and the second crowdsourced annotation data includes fine-tuned second labels corresponding to each first label; for the current iteration state, the number of clusters of the current iteration state is determined based on multiple target annotation clusters of the previous iteration state and the second crowdsourced annotation data, and multiple initial annotation clusters of the current iteration state are determined according to the number of clusters of the current iteration state, and the outlier first labels in the initial annotation clusters of the current iteration state are eliminated to obtain multiple target annotation clusters of the current iteration state, the initial annotation cluster of the current iteration state includes at least one first annotation, and the multiple target annotation clusters of the first iteration state are determined according to the first crowdsourced annotation data and the second crowdsourced annotation data; when the preset stop condition is met, cluster aggregation is performed according to the multiple target annotation clusters of the last iteration state and the second crowdsourced annotation data to obtain a crowdsourcing processing result. It can be seen that in the embodiment of the present application, in the process of determining the number of clusters, full use is made of the fine-tuned second label and the information of the target object itself. Compared with the method of directly taking the maximum number of annotations as the number of clusters in the related art, the number of clusters can be determined more accurately, which provides an important basis for subsequent data processing; and in the clustering process, the outlier first label will be eliminated, which improves the rationality of clustering; finally, by clustering the target annotation cluster and the second crowdsourcing annotation data, the accuracy of the label data is further improved, so that higher quality crowdsourcing processing results can be obtained.

[0028] The method for processing crowdsourced annotation data according to the embodiment of the present application can be executed by an electronic device such as a terminal device or a server, and the terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The method can be implemented by a processor calling a computer-readable program instruction stored in a memory. The server can be an independent physical server, a server cluster composed of multiple servers, or a cloud server capable of cloud computing.

[0029] In a first aspect, an embodiment of the present application provides a method for processing crowdsourced annotation data.

[0030] Figure 1 A flowchart of a method for processing crowdsourced annotation data provided in an embodiment of the present application. Figure 1 , the method for processing crowdsourcing annotation data may include the following steps.

[0031] In step S11, the first crowdsourced annotated data to be processed is processed based on a preset label fine-tuning model to obtain second crowdsourced annotated data.

[0032] The first crowdsourced annotation data includes a first label obtained by multiple annotation subjects annotating the target object. The annotation subject can be a manual annotator or a machine annotator, and the target object is an object to be annotated, which can be an image, etc., which is not limited in the embodiment of the present application. For example, for a pattern including a cat in an image, multiple manual annotators can annotate the cat in the image to obtain multiple first labels of the image, and the multiple first labels constitute the first crowdsourced annotation data.

[0033] Furthermore, after obtaining the first crowd-sourced annotation data, considering that the annotation levels and annotation intentions of different annotation subjects may be different, resulting in uneven quality levels of multiple first labels, the first crowd-sourced annotation data can be processed using the label fine-tuning model to obtain second crowd-sourced annotation data, and the second crowd-sourced annotation data includes fine-tuned second labels corresponding to each first label. In other words, each first label can be fine-tuned through the label fine-tuning model to obtain a second label corresponding to each first label with relatively high accuracy and rationality.

[0034] In some optional implementations, the label fine-tuning model can be a basic visual segmentation model, wherein computer vision segmentation is an important subfield in the field of computer vision. The basic visual segmentation model can be used to assign each pixel in the image to a different category or object.

[0035] Exemplarily, the basic visual segmentation model may be a Segment Anything Model (SAM). Accordingly, if the target object is a target image, the fine-tuning of the first crowdsourced annotation data based on SAM may be characterized as follows:

[0036] A sam =SAM(A,Img)

[0037] Among them, A is the first crowd-sourced annotation data, Img is the target image, and A sam This is the second crowd-sourced labeled data after SAM fine-tuning.

[0038] In step S12, for the current iteration state, the number of clusters of the current iteration state is determined based on the multiple target annotation clusters of the previous iteration state and the second crowdsourced annotation data, and the multiple initial annotation clusters of the current iteration state are determined according to the number of clusters of the current iteration state, and the outlier first labels in the initial annotation clusters of the current iteration state are removed to obtain the multiple target annotation clusters of the current iteration state.

[0039] The initial annotation cluster of the current iteration state includes at least one first annotation, and the multiple target annotation clusters of the first iteration state are determined according to the first crowdsourced annotation data and the second crowdsourced annotation data.

[0040] That is to say, the final target annotation cluster can only be determined through multiple iterative processes, and in each iterative process, the number of clusters is first determined, and then the initial annotation cluster is determined based on the number of clusters, and then the outlier first label is removed to obtain the target annotation cluster of the current iterative process.

[0041] In some optional implementations, the number of clusters in the current iteration state is determined based on multiple target annotation clusters and second crowdsourced annotation data in the previous iteration state, including: constructing multiple first label pairs based on multiple target annotation clusters in the previous iteration state, each first label pair includes two first labels; determining similarity values ​​of each first label pair according to the multiple target annotation clusters in the previous iteration state and the second crowdsourced annotation data; performing connected component analysis according to the similarity values ​​of the multiple first label pairs to obtain multiple first connected components; adjusting the multiple first connected components according to the annotation subject information corresponding to each first label to obtain multiple second connected components; determining the number of clusters in the current iteration state according to the multiple second connected components. Wherein, the similarity value of the first label pair can represent the degree of similarity between the two first labels of the first label pair.

[0042] Exemplarily, the first crowd-sourced annotation data A includes N1 first labels ai (1≤i≤N1). In the first iterative process, multiple first label pairs (am, an) are constructed based on the first crowd-sourced annotation data A, m∈[1,N1], n∈[1,N1], and then the similarity values ​​of the above-mentioned first label pairs are determined according to the first crowd-sourced annotation data and the second crowd-sourced annotation data, and connected component analysis is performed according to the similarity values ​​of the multiple first label pairs to obtain multiple first connected components, and then the multiple first connected components are adjusted according to the annotation subject information corresponding to each first label to obtain multiple second connected components, and finally the number of clusters K1 of the first iterative process is determined according to the multiple second connected components.

[0043] After the number of clusters K1 is obtained, the K1 initial annotation clusters of the first iteration process can be determined by the number of clusters K1, and the outlier first labels are removed to obtain K1 target annotation clusters. It should be noted that since several outlier first labels may be removed, the number of first labels included in the K1 target annotation clusters may be less than N, and in the subsequent iteration process, the corresponding data processing is performed based on the K1 target annotation clusters, that is, the second iteration process is performed based on the remaining first labels after removing the outlier first labels.

[0044] Furthermore, if after the first iteration process, it is determined that the preset stop condition is still not met, the second iteration process is entered.

[0045] In the second iterative process, first, multiple first label pairs (ap, aq) are constructed based on the K1 target annotation clusters, p∈[1,N2], q∈[1,N2], N2 is the total number of first labels remaining after removing outliers; then, the similarity values ​​of the above-mentioned first label pairs are determined according to the N2 first labels corresponding to the K1 target annotation clusters and the second crowdsourcing annotation data, and connected component analysis is performed based on the similarity values ​​of the multiple first label pairs to obtain multiple first connected components, and then the multiple first connected components are adjusted according to the annotation subject information corresponding to each first label to obtain multiple second connected components, and finally, the number of clusters K2 of the second iterative process is determined based on the multiple second connected components.

[0046] Similarly, after the number of clusters K2 is obtained, K2 initial labeling clusters for the second iteration process can be determined by the number of clusters K2, and outlier first labels are removed to obtain K2 target labeling clusters.

[0047] Furthermore, if the preset stop condition is still not satisfied after the second iteration process, the third iteration process is entered. The subsequent iteration processes are similar and will not be described in detail here.

[0048] In some optional implementations, the similarity values ​​of each first label pair are determined based on multiple target label clusters and second crowdsourced label data in a previous iteration state, including: for any first label pair, determining the intersection-and-union ratio of the two first labels corresponding to the first label pair; determining the label quality evaluation value of each first label corresponding to the first label pair; and determining the similarity value of the first label pair based on the intersection-and-union ratio and the label quality evaluation value.

[0049] Among them, the intersection over union (IoU) is the ratio of the intersection and the union of two regions. When the two regions overlap completely, the IoU is the largest and the value is 1; when the two regions do not overlap at all, the IoU is the smallest and the value is 0; the IoU of the two first labels corresponding to the first label pair refers to the ratio between the intersection and the union of the labeled regions corresponding to the two first labels. The label quality evaluation value is a value used to reflect the labeling quality of the first label. If the labeling accuracy of the first label is higher, the value of the label quality evaluation value of the first label is also correspondingly larger. Conversely, if the labeling accuracy of the first label is lower, the value of the label quality evaluation value of the first label is also correspondingly smaller.

[0050] In some optional implementations, considering that a first tag has a corresponding second tag with relatively higher accuracy, the tag quality evaluation value of the first tag can be determined using the second tag corresponding to the first tag.

[0051] In some optional implementations, the tag quality evaluation value may be represented by Formula 1:

[0052]

[0053] Among them, a is the first label, a′ is the second label corresponding to the first label a, σ(a) is the label quality evaluation value of the first label a, IoU(a, a′) represents the intersection over union ratio of the first label a and the second label a′, B represents the minimum enclosing rectangle corresponding to the labeled area corresponding to the first label a and the labeled area corresponding to the second label a′, area(B) represents the area of ​​B, and ∪ is the union symbol.

[0054] In some optional implementations, the similarity value of the first label pair may be determined jointly according to the intersection-over-union ratio of two first labels corresponding to the first label pair and the label quality evaluation values ​​of the two first labels.

[0055] In some optional implementations, the similarity value of the first tag pair may be represented by Formula 2:

[0056] S(a1, a2)=IoU(a1, a1)×σ(a1)×σ(a2) ​​2

[0057] Among them, (a1, a2) represents the first label pair consisting of the first label a1 and the first label a2, S(a1, a2) represents the similarity value of the first label pair (a1, a2), IoU(a1, a2) represents the intersection over union ratio of the first label a1 and the first label a2, σ(a1) is the label quality evaluation value of the first label a1, and σ(a2) ​​is the label quality evaluation value of the first label a2.

[0058] Furthermore, after determining the similarity value of the first tag pair, a connected component analysis may be performed on the plurality of first tags in the current iteration state based on a preset search algorithm, so as to determine the number of clusters in the current iteration state.

[0059] Among them, connected component analysis is usually used to analyze directed graphs. In a directed graph G, if two points are reachable to each other, the two points are said to be strongly connected, and if any two points in G are reachable to each other, G is said to be a strongly connected graph. In an embodiment of the present application, the target object corresponds to a directed graph, and each first label corresponds to a node in the directed graph. Through connected component analysis, the connected components therein can be determined, and each connected component can be regarded as a cluster. In addition, when determining whether two nodes are connected, the main basis is the similarity value of the first label pair, that is, if the similarity value of the first label pair corresponding to the two first labels is high, it is determined that the two first labels are strongly connected. On the contrary, if the similarity value of the first label pair corresponding to the two first labels is low, it is determined that the two first labels are not connected.

[0060] In some optional implementations, connected component analysis is performed based on similarity values ​​of multiple first label pairs to obtain multiple first connected components, including: traversing multiple first label pairs based on the similarity values ​​of the multiple first label pairs and based on a depth-first search algorithm to obtain traversal results; determining multiple first connected components based on the traversal results.

[0061] Exemplarily, if for a target object, the plurality of first label pairs are traversed based on the depth-first search algorithm according to their similarity values, the two first labels corresponding to the first label pairs whose similarity values ​​are greater than or equal to a preset similarity threshold are determined as connected nodes, and the two first labels corresponding to the first label pairs whose similarity values ​​are less than the similarity threshold are determined as disconnected nodes, thereby obtaining a corresponding traversal result. If the traversal result includes several connected subgraphs, several first connected components can be obtained.

[0062] In some optional implementations, after obtaining the first connected component, crowdsourcing information may be further combined to ensure its uniqueness and improve its accuracy and rationality. The crowdsourcing information is information about the labeled subject.

[0063] In some optional implementations, for the first connected component L1, by calculating the similarity value information (such as average similarity value, regional similarity value, etc.) of the second label pairs belonging to the same annotation subject in L1, retaining the first label with a larger similarity value in L1, and deleting the first label with a smaller value, the corresponding second connected component L2 is obtained based on the remaining first labels. In the above processing, the distribution information of the annotation subject is taken into account, so as to avoid the situation where the number of clusters does not conform to the crowdsourcing law.

[0064] In some optional implementations, multiple first connected components are adjusted according to the annotation subject information corresponding to each first label to obtain multiple second connected components, including: for any first connected component, according to the annotation subject information corresponding to each first label in the first connected component, at least one second label pair is obtained, and the two first labels in the second label pair correspond to the same annotation subject; the regional similarity value of each first label in the second label pair is determined, and the regional similarity is used to characterize the degree of similarity between the first label and multiple adjacent first labels; the first label in the first connected component is deleted according to the regional similarity value to obtain the second connected component.

[0065] Further, after the second connected components are determined, the number of clusters can be determined according to the number of the second connected components. For example, if the number of the second connected components is K, the number of clusters is determined to be K.

[0066] In some optional implementations, the process of determining the number of clusters in step S12 can be represented by the following pseudo code:

[0067] Input:Ai: Set of annotations,τ: Similarity threshold

[0068] Output: Number of clusters k

[0069]

[0070] The input data includes Ai and τ, Ai represents the first crowdsourced annotation data about the i-th target image, τ is the similarity threshold (for example, 0.2), and the output data is the number of clusters k.

[0071] The above processing process includes:

[0072] Initialization (Lines 1-2): First, the similarity matrix M is initialized with zero values ​​and the cluster counter k is set to zero, thus preparing the data structures required for subsequent calculations.

[0073] Construct similarity matrix (rows 3-5): For each first label pair (aim, ain) in the first crowdsourced annotation data Ai, calculate the intersection over union (IoU) based on the annotation area, and multiply the intersection over union by the label quality evaluation value (σ) of each first label to obtain the similarity value (score) of the first label pair, and construct similarity matrix M based on multiple similarity values, that is, the value of element Mmn in row m and column n is the similarity value score of the first label pair (aim, ain). In addition, matrix M is equivalent to recording the association relationship between each first label in combination with crowdsourced information.

[0074] Connected component analysis (lines 6-8): A connected component analysis is performed on the similarity matrix M based on a depth-first search (DFS) algorithm to determine multiple first connected components L.

[0075] Combine crowdsourcing information to ensure uniqueness (lines 9-12)}: Using the distribution information of the labeled subjects, the first connected component L is further adjusted to alleviate the situation where the number of clusters does not conform to the crowdsourcing law.

[0076] Return the number of clusters (line 13): Finally, the determined number of clusters k is returned, providing a measure of the granularity of the clustering.

[0077] Among them, the process of connected component analysis based on the depth-first search algorithm (DFS) can be represented by the following pseudo code:

[0078] Input:node:Starting node,M:Similarity matrix,τ:Similarity threshold

[0079] Output:List L of visited L

[0080]

[0081]

[0082] It can be seen from the above pseudo code that the processing process of DFS includes: iterating each first label ain that has not been visited in Ai, and for each unvisited first label, starting the DFS algorithm to traverse to identify all connected first labels. During the traversal process, a similarity threshold is passed. If the similarity value (score) corresponding to the first label pair composed of two first labels exceeds the similarity threshold, the two are considered to be connected. Finally, these connected first labels form a labeled cluster. It should be noted that compared with the related art, this method integrates crowdsourcing information by considering the label distribution between the labeling subjects in the entire solution process, so it helps to improve the clustering process, thereby determining a more accurate number of clusters.

[0083] Furthermore, the process of adjusting the first connected component using crowdsourcing information can be represented by the following pseudo code:

[0084] Input:L:List of visited nodes,M:Similarity matrix

[0085]

[0086] It can be seen from the above pseudo code that the above processing process includes: calculating the regional similarity values ​​of each first label of the second label pair belonging to the same annotation subject in L, and selecting the first label with the largest regional similarity value in the second label pair as the supporting factor for the number of clusters, which takes into account the distribution information of the annotation subject, and can alleviate the situation where the number of clusters does not conform to the crowdsourcing law.

[0087] The regional similarity value may be the average of similarity values ​​of multiple first label pairs related to the first label in the similarity matrix M. For example, the regional similarity value of the first label is obtained by calculating the average of multiple similarity values ​​corresponding to the first label pairs containing the first label in M. For another example, the regional similarity value is determined according to the average of multiple similarity values ​​corresponding to the first label pairs in the row or column containing the first label in M.

[0088] In some optional implementations, for any iteration state, after the corresponding number of clusters is determined, a corresponding plurality of initial labeled clusters may be determined according to the number of clusters.

[0089] In some optional implementations, multiple initial annotation clusters may be determined based on spectral clustering.

[0090] Exemplarily, multiple initial annotation clusters for the current iteration state are determined based on the number of clusters in the current iteration state, including: determining multiple candidate subgraphs based on the number of clusters in the current iteration state, each candidate subgraph including at least one first label; determining a subgraph quality assessment value and an annotation body quality assessment value of each candidate subgraph; determining whether the multiple candidate subgraphs meet preset cutting conditions based on the subgraph quality assessment values ​​and the annotation body quality assessment values ​​of the multiple candidate subgraphs; and determining multiple initial annotation clusters for the current iteration state based on the multiple candidate subgraphs when the preset cutting conditions are met.

[0091] In some optional implementations, the sub-image quality evaluation value is an average of the label quality evaluation values ​​of multiple first labels in the candidate sub-image.

[0092] In some optional implementations, for a certain annotation subject w j , its annotation subject quality assessment value η(w j ) can be represented by the following formula:

[0093]

[0094] Among them, w j represents the jth annotation subject about the target object A, |A j | indicates the annotation subject w j The number of all first labels annotated for the target object A, aji is the i-th first label annotated by the annotation subject for the target object, i∈[1,|A j|], σ(aji) represents the label quality evaluation value of the first label aji.

[0095] Correspondingly, the annotation subject quality evaluation value of the candidate subgraph may include the annotation subject quality evaluation values ​​of all annotation subjects corresponding to the first label in the candidate subgraph.

[0096] In some optional implementations, the first crowd-sourced labeled data may be clustered according to the number of clusters based on a crowd-sourced cutting method. The crowd-sourced cutting method may be characterized as the following optimization function:

[0097]

[0098] Among them, C i represents the i-th candidate subgraph, corresponding to an initial labeled cluster, i∈[1,k], k is the number of clusters, σ(C i ) means C i The sub-image quality evaluation value is Represents C i The corresponding annotation subject quality assessment value, To remove C i When the above optimization function takes the minimum value (equivalent to satisfying the preset cutting condition), it means that the current cutting method is a better cutting method. i , which has a weak correlation with the remaining sub-graphs, C i It can be used as a candidate subgraph that meets the preset cutting condition, so that multiple first labels are divided into k candidate subgraphs, each candidate subgraph corresponds to an initial annotation cluster.

[0099] It should be noted that by cutting the sample point connection matrix (i.e., a matrix composed of multiple first labels) in this cutting manner, crowdsourcing information can be fully utilized for clustering, so that each subgraph (or cluster) has a certain annotation subject quality and label quality as support.

[0100] The theoretical proof process of the feasibility of the above crowdsourcing cutting method is as follows:

[0101] Assume L is the Laplacian matrix of an undirected graph and f is R n Any vector in . The total variation of the graph signal represented by f is:

[0102]

[0103] Among them, s ij represents the weight of the edge between nodes i and j, f i represents the i-th component of f.

[0104] In order to minimize the value of CrowdCut, the indicator vector h is introduced i ∈{h1,h2,...,h k}, i = 1, 2, ..., k, for any vector h j , is an n-dimensional vector (n is the number of samples), and h is defined ij for:

[0105]

[0106] Then,

[0107]

[0108] For subgraph i, its CrowdCut corresponds to Therefore, the CrowdCut function expression of k subgraphs is:

[0109]

[0110] At this time, H T MH=I, the proof is as follows:

[0111]

[0112] Because h mi and h ni cannot be non-zero at the same time (v i ∈V m and v i ∈V n ), the off-diagonal elements are 0 and the diagonal elements are 1. Therefore, the optimization objective becomes:

[0113]

[0114] H T MH=I

[0115] make The optimization objective becomes:

[0116]

[0117] F T F=I

[0118] Each indicator vector in the matrix H is n-dimensional, where each variable in the vector is either 0 or Therefore, each indicator vector has 2 n possible values. For k subgraphs, there are k×2 n There are a lot of possible H matrices, making it an NP-hard problem to find an H that satisfies the above optimization goal.

[0119] Furthermore, we can use the idea of ​​spectral clustering to approximately solve this NP-hard problem. The goal is to find The k smallest eigenvalues ​​of . Usually, k is much smaller than n, so it is equivalent to a dimensionality reduction process, which approximately solves the NP-hard problem.

[0120] In some optional implementations, after the initial annotation cluster is obtained, the outlier first labels (which may be referred to as outlier labels) may be removed to further improve the accuracy and rationality of the initial annotation cluster.

[0121] In some optional implementations, removing outlier first labels in an initial annotation cluster of a current iteration state includes: determining the outlier probability of each first label in the initial annotation cluster; determining the outlier first labels in the initial annotation cluster of the current iteration state according to the outlier probability; and removing the outlier first labels from the initial annotation cluster.

[0122] Exemplarily, for each first label in the initial annotation cluster, the outlier probability of the first label can be calculated in combination with the annotation distribution and the annotation subject information. When the outlier probability is large, it is determined that the first label belongs to the outlier first label and needs to be removed. When the outlier probability is small, the first label is retained.

[0123] In some optional implementations, the outlier probability p of the first tag can be represented by the following formula:

[0124]

[0125] and,

[0126] Among them, i is the serial number of the initial annotation cluster, j is the serial number of the first label in the i-th initial annotation cluster, and a ij represents the jth first label in the i initial annotation clusters, a mn represents the nth first label in the m initial annotation clusters, |C i | is the initial labeled cluster C i The number of the first label in area(a mn ) means a mn The area of ​​the marked area, ∩ is the intersection symbol, W(a mn ) indicates the first label a mn The corresponding annotation subject, w refers to the annotation subject in general, W(C i ) represents the initial labeled cluster C i The corresponding annotation subject, Used to determine the initial annotation cluster C i The first label a mn The number of first labels annotated by the annotation subject, σ(aij ) is the first label a ij The label quality evaluation value of (Condition) represents an indicator function. When the condition is met, its value is 1, and when the condition is not met, its value is 0. argmax(f(x)) is the variable point x or the set of x corresponding to the maximum value of f(x). category represents the label type. Used to determine the initial annotation cluster C i The label type c that maximizes the sum of label quality evaluation values i .

[0127] Furthermore, after obtaining the outlier probability of the first label, the outlier probability is compared with a preset outlier probability threshold. If the outlier probability is greater than or equal to the preset outlier probability threshold, the first label is removed from the initial annotation cluster where the first label is located. If the outlier probability is less than the preset outlier probability threshold, the first label is retained.

[0128] In step S13, when a preset stop condition is met, cluster aggregation is performed according to multiple target annotation clusters in the last iteration state and the second crowdsourcing annotation data to obtain a crowdsourcing processing result.

[0129] In some optional implementations, the preset stop condition is a condition for determining whether to stop iteration, which may include the number of iterations exceeding a preset threshold, etc., which is not limited in the embodiments of the present application.

[0130] It can be seen that when the preset stop condition is met, the multiple target annotation clusters in the last iteration state are taken as the final annotation clusters, and cluster aggregation is performed in combination with the second crowdsourcing annotation data to obtain the crowdsourcing processing result.

[0131] In some optional implementations, the first label corresponds to a bounding box and a label type, and the label type is used to reflect the type of the label; accordingly, cluster aggregation is performed based on the multiple target annotation clusters and the second crowdsourcing annotation data of the last iteration state to obtain a crowdsourcing processing result, including: for each target annotation cluster of the last iteration state, weighted aggregation is performed on the bounding boxes of the multiple first labels in the target annotation cluster and the bounding boxes of the second labels corresponding to the multiple first labels to obtain a bounding box aggregation result; type aggregation is performed based on the label types of the multiple first labels in the target annotation cluster to obtain a type aggregation result; and the crowdsourcing processing result is determined based on the bounding box aggregation result and the type aggregation result.

[0132] It can be seen that when performing aggregation processing, aggregation can be performed from the two dimensions of bounding box and label type, and the crowdsourcing processing results obtained include the aggregated label bounding box and the aggregated label type. In addition, since the second crowdsourcing annotation data with relatively high accuracy is used in the aggregation process, it is equivalent to aggregating this accuracy into the crowdsourcing processing results, thereby effectively improving the accuracy of the crowdsourcing processing results.

[0133] In some optional implementations, the aggregation process of the bounding box can be represented as follows:

[0134]

[0135] Among them, a ij Label the jth first label in the i-th target cluster, w(a ij ) is the first label a ij The annotation subject, η(w(a ij )) represents the annotation subject w(a ij )’s annotation subject quality assessment value, r ij Indicates the first label a ij The bounding box of Indicates that the first label a ij The corresponding bounding box of the second label, |C i | represents the number of the first label of the i-th target annotation cluster, Represents the target annotation cluster C i The aggregated bounding box.

[0136] In some optional implementations, the tag type aggregation process can be represented by the following formula:

[0137]

[0138] Among them, category indicates the tag type. Used to determine the target annotation cluster C i The label quality evaluation value σ(a ij ) and the annotation subject quality evaluation value η(w(a ij ))) is the label type with the largest product, which is the target label cluster C i The aggregated tag type

[0139] Based on this, the crowdsourcing processing results obtained include multiple target annotation clusters, each of which includes an aggregated bounding box for characterizing the annotation area and corresponds to an aggregated label type.

[0140] In the embodiment provided by the present application, the first crowd-sourced annotation data to be processed is processed based on a preset label fine-tuning model to obtain second crowd-sourced annotation data, wherein the first crowd-sourced annotation data includes first labels obtained by multiple annotation subjects annotating target objects, and the second crowd-sourced annotation data includes fine-tuned second labels corresponding to each first label; for the current iteration state, the number of clusters of the current iteration state is determined based on multiple target annotation clusters of the previous iteration state and the second crowd-sourced annotation data, multiple initial annotation clusters of the current iteration state are determined according to the number of clusters of the current iteration state, and outlier first labels in the initial annotation clusters of the current iteration state are eliminated to obtain multiple target annotation clusters of the current iteration state, wherein the initial annotation cluster of the current iteration state includes at least one first annotation, and the multiple target annotation clusters of the first iteration state are determined according to the first crowd-sourced annotation data and the second crowd-sourced annotation data; when the preset stop condition is met, cluster aggregation is performed according to the multiple target annotation clusters of the last iteration state and the second crowd-sourced annotation data to obtain a crowd-sourcing processing result. It can be seen that in the embodiment of the present application, in the process of determining the number of clusters, full use is made of the fine-tuned second label and the information of the target object itself. Compared with the method of directly taking the maximum number of annotations as the number of clusters in the related art, the number of clusters can be determined more accurately, which provides an important basis for subsequent data processing; and in the clustering process, the outlier first label will be eliminated, which improves the rationality of clustering; finally, by clustering the target annotation cluster and the second crowdsourcing annotation data, the accuracy of the label data is further improved, so that higher quality crowdsourcing processing results can be obtained.

[0141] Figure 2 A schematic diagram of the processing process of a method for processing crowdsourced annotation data provided in Example 1 of the present application. Figure 2 , the target object is an image, and after the target object is annotated by multiple annotation subjects, first crowd-sourced annotation data including multiple first labels are obtained. On this basis, the first label in the first crowd-sourced annotation data is fine-tuned using the label fine-tuning model to obtain second crowd-sourced annotation data, and the second crowd-sourced annotation data includes fine-tuned second labels corresponding to each first label. Moreover, since the second label is a fine-tuned label, the accuracy and rationality of the second label are improved compared to the first label, which may be reflected in the distribution position and size of the annotation area being more accurate and reasonable.

[0142] After obtaining the first crowd-sourced labeled data and the second crowd-sourced labeled data, label clustering can be performed. Figure 2As shown, the label clustering process is an iterative process, and in each iteration process, operations such as determining the number of clusters, spectral clustering, and outlier processing need to be performed. Among them, determining the number of clusters is mainly to determine the number of clusters used for label cluster division. Spectral clustering is to cluster the first label to be clustered based on the determined number of clusters to obtain multiple initial label clusters. Outlier processing is to determine the outlier labels in the initial label cluster and remove them from the initial label cluster to obtain the target label cluster. In the next iteration process, the multiple first labels corresponding to the multiple target label clusters determined in the previous iteration process are used as labels to be clustered to perform a new round of iterative processing. After multiple rounds of iteration, if the preset stop condition is met, the iteration is stopped, and the target label cluster determined by the last iteration process is used as the final label clustering result.

[0143] After completing the label clustering, you can enter the label aggregation process. Specifically, in the final label clustering result, each target label cluster may include multiple first labels, the distribution position and size of the label area of ​​each first label may be different, and the label category of the first label may also be inconsistent. Based on this, in order to improve the accuracy of the labeling result, the multiple first labels in each target label cluster can be aggregated with the help of the second label, so as to obtain a more accurate and reasonable label label corresponding to each target label cluster.

[0144] Among them, label aggregation mainly includes two types of aggregation processing methods. The first type of aggregation processing method is bounding box aggregation, that is, aggregating the bounding boxes of labels to merge the bounding boxes of multiple labels into a more accurate and reasonable aggregated bounding box. The second type of aggregation processing method is category aggregation, that is, aggregating label types to obtain more accurate and reasonable label types. Through bounding box aggregation and category aggregation, the aggregated bounding box and aggregated label type of each target annotation cluster can be obtained, thereby obtaining the final crowdsourcing processing result.

[0145] Figure 3 A schematic diagram of the processing process of a method for processing crowdsourced annotation data provided in Example 2 of the present application. Figure 3 The processing process of the crowd-sourced annotation data mainly includes: firstly obtaining the first crowd-sourced annotation data about the target object (for example, a certain image), and using the label fine-tuning model to fine-tune the first crowd-sourced annotation data to obtain the second crowd-sourced annotation data, and then determining the number of clusters in turn on this basis, and performing label clustering based on the determined number of clusters, and proposing outlier labels in the cluster, and then obtaining the final crowd-sourced processing result through aggregation processing. Among them, determining the number of clusters, label clustering, and removing outlier labels can be performed iteratively, that is, in each iteration process, the above operations are performed in turn, and the iteration is stopped when it is determined that the preset stop condition is met, and aggregation processing is performed on the basis of the current iteration result to obtain the crowd-sourced processing result.

[0146] Furthermore, in each iteration process, when determining the number of clusters, a connected component analysis is performed on multiple first labels in combination with the target objects to obtain multiple first connected components, and then the first connected components are adjusted in combination with the labeled subject information to obtain multiple second connected components, and then the number of clusters is determined based on the multiple second connected components.

[0147] During label clustering, for a similarity matrix formed based on similarity values ​​of the first label, multiple clusters may be determined according to the obtained number of clusters by using spectral clustering, and each cluster corresponds to an initial labeled cluster.

[0148] For each initial annotation cluster, the degree of belonging between the first label and the initial annotation cluster to which it belongs can be judged through the area of ​​the annotation area of ​​the first label, the distribution position, the label quality evaluation value, the label category and the annotation subject information, so as to obtain the outlier probability of the first label, and then the first label with a higher outlier probability can be removed from the initial annotation cluster to which it belongs as an outlier label.

[0149] Figure 4 A flowchart of a method for processing crowdsourced annotation data provided in an embodiment of the present application. Figure 4 , the method for processing crowdsourcing annotation data may include the following steps.

[0150] Step S401: Acquire first crowdsourced annotation data about a target object, where the first crowdsourced annotation data includes a plurality of first tags.

[0151] Step S402: fine-tune the first crowd-sourced annotation data based on the label fine-tuning model to obtain second crowd-sourced annotation data, where the second crowd-sourced annotation data includes fine-tuned second labels corresponding to each first label.

[0152] Step S403, during the first iteration process, the number of clusters of the first iteration state is determined according to the multiple first labels of the first crowdsourced annotation data and the second crowdsourced annotation data, the multiple initial annotation clusters of the first iteration state are determined according to the number of clusters of the first iteration state, and the outlier first labels in the initial annotation clusters of the first iteration state are removed to obtain the multiple target annotation clusters of the first iteration state.

[0153] Step S404, determining whether a preset stop condition is met.

[0154] If it is determined that the preset stop condition is not met, jump to step S405; if it is determined that the preset stop condition is met, jump to step S407.

[0155] Step S405, updating the iteration status.

[0156] For example, if the previous iteration state corresponds to the z-1th iteration process, the current iteration state can be updated to the zth iteration state, z≥2.

[0157] Step S406, during the zth iteration process, the number of clusters of the zth iteration state is determined based on the multiple target annotation clusters of the z-1th iteration state and the second crowdsourced annotation data, and the multiple initial annotation clusters of the zth iteration state are determined according to the number of clusters of the zth iteration state, and the outlier first labels in the initial annotation clusters of the zth iteration state are removed to obtain multiple target annotation clusters of the zth iteration state, z≥2.

[0158] After executing step S406, jump to step S404.

[0159] Step S407 : for each target annotation cluster in the last iteration state, weighted aggregation is performed on the bounding boxes of the multiple first labels in the target annotation cluster and the bounding boxes of the second labels corresponding to the multiple first labels to obtain a bounding box aggregation result.

[0160] Step S408 , performing type aggregation according to the tag types of the plurality of first tags in the target annotation cluster to obtain a type aggregation result.

[0161] Step S409: determining a crowdsourcing processing result according to the bounding box aggregation result and the type aggregation result.

[0162] In summary, the embodiment of the present application introduces a label fine-tuning model to integrate the feature information of crowdsourcing labeling results, proposes a depth-first search algorithm combined with crowdsourcing information to determine the number of clustering clusters, and a new spectral clustering method based on crowdsourcing cutting, which effectively clusters the labeling results, combines the labeling subject intention and distribution information to remove outliers, effectively reduces clustering noise, and estimates the quality of the labeling subject and the labeling result quality based on the feature information, effectively improving the accuracy of multi-target and multi-category complex label aggregation results.

[0163] In short, the embodiments of the present application combine the information of the target object (such as image information) by introducing a basic computer vision segmentation model (i.e., a label fine-tuning model) and a new spectral clustering method based on crowdsourcing cutting, which effectively captures the relationship between the annotation results, improves the accuracy of aggregation, and optimizes the aggregation process of complex label crowdsourcing results.

[0164] It can be understood that the above-mentioned various method embodiments mentioned in this application can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, this application will not go into details. Those skilled in the art can understand that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined by its function and possible internal logic.

[0165] In a second aspect, an embodiment of the present application provides a device for processing crowdsourced annotation data.

[0166] Figure 5 A block diagram of a crowdsourcing annotation data processing device provided in an embodiment of the present application. Figure 5 The crowdsourcing annotation data processing device 500 may include the following modules.

[0167] A fine-tuning module 501 is used to process the first crowd-sourced annotation data to be processed based on a preset label fine-tuning model to obtain second crowd-sourced annotation data, wherein the first crowd-sourced annotation data includes first labels obtained by multiple annotation subjects annotating the target object, and the second crowd-sourced annotation data includes fine-tuned second labels corresponding to each first label;

[0168] A clustering module 502 is used to determine the number of clusters of the current iteration state based on the multiple target annotation clusters of the previous iteration state and the second crowdsourced annotation data for the current iteration state, determine multiple initial annotation clusters of the current iteration state according to the number of clusters of the current iteration state, and remove the outlier first label in the initial annotation cluster of the current iteration state to obtain multiple target annotation clusters of the current iteration state, wherein the initial annotation cluster of the current iteration state includes at least one first annotation, and the multiple target annotation clusters of the first iteration state are determined according to the first crowdsourced annotation data and the second crowdsourced annotation data;

[0169] Aggregation module 503 is used to perform cluster aggregation based on multiple target annotation clusters and second crowdsourcing annotation data in the last iteration state to obtain a crowdsourcing processing result when a preset stop condition is met.

[0170] In some optional implementations, the number of clusters in the current iteration state is determined based on multiple target annotation clusters in the previous iteration state and the second crowdsourced annotation data, including: constructing multiple first label pairs based on the multiple target annotation clusters in the previous iteration state, each first label pair including two first labels; determining similarity values ​​of each first label pair according to the multiple target annotation clusters in the previous iteration state and the second crowdsourced annotation data; performing connected component analysis based on the similarity values ​​of the multiple first label pairs to obtain multiple first connected components; adjusting the multiple first connected components according to the annotation subject information corresponding to each first label to obtain multiple second connected components; and determining the number of clusters in the current iteration state based on the multiple second connected components.

[0171] In some optional implementations, connected component analysis is performed based on similarity values ​​of multiple first label pairs to obtain multiple first connected components, including: traversing multiple first label pairs based on the similarity values ​​of the multiple first label pairs and based on a depth-first search algorithm to obtain traversal results; determining multiple first connected components based on the traversal results.

[0172] In some optional implementations, the similarity values ​​of each first label pair are determined based on multiple target label clusters and second crowdsourced label data in a previous iteration state, including: for any first label pair, determining the intersection-and-union ratio of the two first labels corresponding to the first label pair; determining the label quality evaluation value of each first label corresponding to the first label pair; and determining the similarity value of the first label pair based on the intersection-and-union ratio and the label quality evaluation value.

[0173] In some optional implementations, the tag quality evaluation value may be represented by Formula 1:

[0174] σ(a)="IoU"(a,a^')-(area(B)-(a∪a^')) / (area(B)) 1

[0175] Where a is the first label, a^' is the second label corresponding to the first label a, σ(a) is the label quality evaluation value of the first label a, "IoU"(a,a^') represents the intersection over union ratio of the first label a and the second label a^', B represents the minimum enclosing rectangle of the labeled area corresponding to the first label a and the labeled area corresponding to the second label a^', area(B) represents the area of ​​B, and ∪ is the union symbol;

[0176] The similarity value of the first label pair can be represented by Formula 2:

[0177] S(a1,a2)="IoU"(a1,a1)×σ(a1)×σ(a2) ​​2

[0178] Wherein, (a1, a2) represents the first label pair consisting of the first label a1 and the first label a2, S(a1, a2) represents the similarity value of the first label pair (a1, a2), "IoU"(a1, a2) represents the intersection over union ratio of the first label a1 and the first label a2, σ(a1) is the label quality evaluation value of the first label a1, and σ(a2) ​​is the label quality evaluation value of the first label a2.

[0179] In some optional implementations, multiple first connected components are adjusted according to the annotation subject information corresponding to each first label to obtain multiple second connected components, including: for any first connected component, according to the annotation subject information corresponding to each first label in the first connected component, at least one second label pair is obtained, and the two first labels in the second label pair correspond to the same annotation subject; the regional similarity value of each first label in the second label pair is determined, and the regional similarity is used to characterize the degree of similarity between the first label and multiple adjacent first labels; the first label in the first connected component is deleted according to the regional similarity value to obtain the second connected component.

[0180] In some optional implementations, multiple initial annotation clusters for the current iteration state are determined based on the number of clusters in the current iteration state, including: determining multiple candidate subgraphs based on the number of clusters in the current iteration state, each candidate subgraph including at least one first label; determining a subgraph quality assessment value and an annotation body quality assessment value of each candidate subgraph; determining whether the multiple candidate subgraphs meet preset cutting conditions based on the subgraph quality assessment values ​​and the annotation body quality assessment values ​​of the multiple candidate subgraphs; and determining multiple initial annotation clusters for the current iteration state based on the multiple candidate subgraphs when the preset cutting conditions are met.

[0181] In some optional implementations, removing outlier first labels in an initial annotation cluster of a current iteration state includes: determining the outlier probability of each first label in the initial annotation cluster; determining the outlier first labels in the initial annotation cluster of the current iteration state according to the outlier probability; and removing the outlier first labels from the initial annotation cluster.

[0182] In some optional implementations, the first label corresponds to a bounding box and a label type; accordingly, the aggregation module 503 performs cluster aggregation based on the multiple target annotation clusters and the second crowdsourcing annotation data of the last iteration state, and when obtaining the crowdsourcing processing result, the following steps are performed: for each target annotation cluster of the last iteration state, weighted aggregation is performed on the bounding boxes of the multiple first labels in the target annotation cluster and the bounding boxes of the second labels corresponding to the multiple first labels to obtain a bounding box aggregation result; type aggregation is performed according to the label types of the multiple first labels in the target annotation cluster to obtain a type aggregation result; and the crowdsourcing processing result is determined based on the bounding box aggregation result and the type aggregation result.

[0183] In the embodiment provided by the present application, a fine-tuning module is used to process the first crowd-sourced annotation data to be processed based on a preset label fine-tuning model to obtain second crowd-sourced annotation data, wherein the first crowd-sourced annotation data includes first labels obtained by multiple annotation subjects annotating the target object, and the second crowd-sourced annotation data includes fine-tuned second labels corresponding to each first label; a clustering module is used to determine the number of clusters of the current iteration state based on multiple target annotation clusters of the previous iteration state and the second crowd-sourced annotation data, and multiple initial annotation clusters of the current iteration state are determined according to the number of clusters of the current iteration state, and outlier first labels in the initial annotation clusters of the current iteration state are eliminated to obtain multiple target annotation clusters of the current iteration state, wherein the initial annotation cluster of the current iteration state includes at least one first annotation, and the multiple target annotation clusters of the first iteration state are determined according to the first crowd-sourced annotation data and the second crowd-sourced annotation data; and an aggregation module is used to perform cluster aggregation according to the multiple target annotation clusters of the last iteration state and the second crowd-sourced annotation data when a preset stop condition is met to obtain a crowd-sourcing processing result. It can be seen that in the embodiment of the present application, in the process of determining the number of clusters, full use is made of the fine-tuned second label and the information of the target object itself. Compared with the method of directly taking the maximum number of annotations as the number of clusters in the related art, the number of clusters can be determined more accurately, which provides an important basis for subsequent data processing; and in the clustering process, the outlier first label will be eliminated, which improves the rationality of clustering; finally, by clustering the target annotation cluster and the second crowdsourcing annotation data, the accuracy of the label data is further improved, so that higher quality crowdsourcing processing results can be obtained.

[0184] Each module in the above-mentioned crowd-sourced annotation data processing device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0185] In addition, the present application also provides electronic devices and computer-readable storage media, which can be used to implement any crowdsourcing annotation data processing method provided in the present application. The corresponding technical solutions and descriptions are referred to in the corresponding records of the method part and will not be repeated here.

[0186] Figure 6 A block diagram of an electronic device provided in an embodiment of the present application.

[0187] Reference Figure 6An embodiment of the present application provides an electronic device, which includes: at least one processor 601; at least one memory 602, and one or more I / O interfaces 603, connected between the processor 601 and the memory 602; wherein the memory 602 stores one or more computer programs that can be executed by at least one processor 601, and the one or more computer programs are executed by at least one processor 601 so that the at least one processor 601 can execute the above-mentioned crowdsourcing annotation data processing method.

[0188] Each module in the above electronic device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0189] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored, wherein the computer program implements the above-mentioned method for processing crowdsourced annotation data when executed by a processor. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0190] An embodiment of the present application also provides a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above-mentioned method for processing crowdsourcing annotation data.

[0191] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable storage medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium).

[0192] As is known to those of ordinary skill in the art, the term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable program instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technology, portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. In addition, it is known to those of ordinary skill in the art that communication media typically contain computer-readable program instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery medium.

[0193] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0194] The computer program instructions for performing the operation of the present application can be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions can be executed completely on a user's computer, partially on a user's computer, executed as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet). In some embodiments, by using the state information of a computer-readable program instruction to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA) or a programmable logic array (PLA), the electronic circuit can execute a computer-readable program instruction, thereby realizing various aspects of the present application.

[0195] The computer program product described herein may be implemented in hardware, software, or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK), etc.

[0196] Various aspects of the present application are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0197] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0198] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0199] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can also be executed in the opposite order sometimes, depending on the function involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a dedicated hardware-based system that performs the function or action of the specification, or can be realized by a combination of special-purpose hardware and computer instructions.

[0200] Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for limiting purposes. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly noted, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, those skilled in the art will appreciate that various changes in form and detail may be made without departing from the scope of the present application as set forth in the appended claims.

Claims

1. A method for processing crowdsourced annotation data, characterized in that: The method comprises: The first crowd-sourced annotation data to be processed is processed based on a preset label fine-tuning model to obtain second crowd-sourced annotation data, wherein the first crowd-sourced annotation data includes first labels obtained by multiple annotation subjects annotating the target object, and the second crowd-sourced annotation data includes fine-tuned second labels corresponding to each of the first labels; For the current iteration state, the number of clusters of the current iteration state is determined based on the multiple target annotation clusters of the previous iteration state and the second crowdsourced annotation data, multiple initial annotation clusters of the current iteration state are determined according to the number of clusters of the current iteration state, and the outlier first labels in the initial annotation clusters of the current iteration state are removed to obtain multiple target annotation clusters of the current iteration state, wherein the initial annotation clusters of the current iteration state include at least one first annotation, and the multiple target annotation clusters of the first iteration state are determined according to the first crowdsourced annotation data and the second crowdsourced annotation data; When a preset stop condition is met, cluster aggregation is performed according to the multiple target annotation clusters in the last iteration state and the second crowdsourcing annotation data to obtain a crowdsourcing processing result.

2. The method according to claim 1, characterized in that The determining the number of clusters in the current iteration state based on the multiple target annotation clusters in the previous iteration state and the second crowdsourced annotation data includes: Based on multiple target annotation clusters in the previous iteration state, construct multiple first label pairs, each first label pair includes two first labels; Determine a similarity value of each first label pair according to a plurality of target label clusters in a previous iteration state and the second crowdsourced label data; Performing connected component analysis according to similarity values ​​of the plurality of first label pairs to obtain a plurality of first connected components; Adjusting the first connected components according to the labeled subject information corresponding to each first tag to obtain a plurality of second connected components; The number of clusters in a current iteration state is determined according to the plurality of second connected components.

3. The method according to claim 2, characterized in that The method of performing connected component analysis according to the similarity values ​​of the plurality of first tag pairs to obtain the plurality of first connected components includes: According to the similarity values ​​of the plurality of first label pairs, the plurality of first label pairs are traversed based on a depth-first search algorithm to obtain a traversal result; A plurality of the first connected components are determined according to the traversal result.

4. The method according to claim 2, characterized in that: The determining, according to the plurality of target annotation clusters in the previous iteration state and the second crowdsourced annotation data, the similarity value of each first tag pair comprises: For any first label pair, determining an intersection-over-union ratio of two first labels corresponding to the first label pair; Determining a label quality assessment value of each first label corresponding to the first label pair; A similarity value of the first label pair is determined according to the intersection-over-union ratio and the label quality evaluation value.

5. The method according to claim 4, characterized in that The label quality evaluation value is represented by Formula 1: Among them, a is the first label, a ′ is the second label corresponding to the first label a, σ(a) is the label quality evaluation value of the first label a, IoU(a,a ′ ) represents the first label a and the second label a ′ The intersection-and-union ratio of the first label a and the second label a ′ The minimum enclosing rectangle corresponding to the corresponding marked area, area(B) represents the area of ​​B, and ∪ is the union symbol; The similarity value of the first tag pair is represented by Formula 2: S(a1,a2)=IoU(a1,a1)×σ(a1)×σ(a2)2 Wherein, (a1, a2) represents the first label pair consisting of the first label a1 and the first label a2, S(a1, a2) represents the similarity value of the first label pair (a1, a2), IoU(a1, a2) represents the intersection over union ratio of the first label a1 and the first label a2, σ(a1) is the label quality evaluation value of the first label a1, and σ(a2) ​​is the label quality evaluation value of the first label a2.

6. The method according to claim 2, characterized in that The step of adjusting the plurality of first connected components according to the labeled subject information corresponding to each first tag to obtain a plurality of second connected components includes: For any one of the first connected components, obtaining at least one second label pair according to the label subject information corresponding to each first label in the first connected component, wherein the two first labels in the second label pair correspond to the same label subject; Determine a region similarity value of each first tag in the second tag pair, wherein the region similarity is used to characterize a degree of similarity between the first tag and a plurality of adjacent first tags; The first label in the first connected component is deleted according to the region similarity value to obtain the second connected component.

7. The method according to claim 1, characterized in that The step of determining a plurality of initial labeled clusters of the current iteration state according to the number of clusters of the current iteration state includes: Determine a plurality of candidate subgraphs according to the number of clusters in the current iteration state, each candidate subgraph including at least one first label; Determining a sub-image quality assessment value and a labeling subject quality assessment value of each of the candidate sub-images; Determining whether the plurality of candidate subgraphs meet a preset cutting condition according to the subgraph quality evaluation values ​​and the annotation subject quality evaluation values ​​of the plurality of candidate subgraphs; When the preset cutting condition is met, a plurality of initial labeled clusters in the current iteration state are determined according to the plurality of candidate subgraphs.

8. The method according to claim 1, characterized in that The step of removing the outlier first label in the initial annotation cluster in the current iteration state includes: Determining the outlier probability of each first label in the initial annotation cluster; Determine, according to the outlier probability, a first label of an outlier in the initial annotation cluster of the current iteration state; The outlier first label is removed from the initial labeled cluster.

9. The method according to claim 1, characterized in that: The first label corresponds to a bounding box and a label type; The clustering is performed according to the multiple target annotation clusters in the last iteration state and the second crowdsourcing annotation data to obtain a crowdsourcing processing result, including: For each target annotation cluster in the last iteration state, weightedly aggregate bounding boxes of multiple first labels in the target annotation cluster and bounding boxes of second labels corresponding to the multiple first labels to obtain a bounding box aggregation result; Perform type aggregation according to the tag types of the multiple first tags in the target annotation cluster to obtain a type aggregation result; The crowdsourcing processing result is determined according to the bounding box aggregation result and the type aggregation result.

10. A device for processing crowdsourced annotation data, characterized in that: The device comprises: A fine-tuning module, configured to process the first crowd-sourced annotation data to be processed based on a preset label fine-tuning model to obtain second crowd-sourced annotation data, wherein the first crowd-sourced annotation data includes first labels obtained by multiple annotation subjects annotating the target object, and the second crowd-sourced annotation data includes fine-tuned second labels corresponding to each of the first labels; A clustering module, for determining the number of clusters of the current iteration state based on the multiple target annotation clusters of the previous iteration state and the second crowdsourced annotation data for the current iteration state, determining multiple initial annotation clusters of the current iteration state according to the number of clusters of the current iteration state, and removing outlier first labels in the initial annotation clusters of the current iteration state to obtain multiple target annotation clusters of the current iteration state, wherein the initial annotation clusters of the current iteration state include at least one first annotation, and the multiple target annotation clusters of the first iteration state are determined according to the first crowdsourced annotation data and the second crowdsourced annotation data; The aggregation module is used to perform cluster aggregation based on multiple target annotation clusters in the last iteration state and the second crowdsourcing annotation data to obtain a crowdsourcing processing result when a preset stop condition is met.

11. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, and the one or more computer programs are executed by the at least one processor so that the at least one processor can execute the method for processing crowdsourced annotation data as described in any one of claims 1-9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the method for processing crowdsourced annotation data according to any one of claims 1 to 9.

13. A computer program product, characterized in that The invention comprises a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the method for processing crowdsourcing annotation data as described in any one of claims 1 to 9.

Images