Triplet task set construction method and apparatus, and electronic device
By constructing a difficulty evaluation index parameter based on the minimum angle of a triangle, the triplet tasks are sorted and grouped for sampling, which solves the redundancy problem in the triplet task set and improves task diversity and model training effect.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NETEASE (HANGZHOU) NETWORK CO LTD
- Filing Date
- 2022-11-16
- Publication Date
- 2026-07-10
AI Technical Summary
Existing active learning strategies for triplet tasks suffer from redundant sample selection and insufficient diversity strategy design in crowdsourcing platforms, leading to a decline in model performance.
By obtaining the difficulty evaluation index parameters of each task in the unlabeled triple task set, a triangle for evaluating the difficulty of task labeling is constructed. The difficulty evaluation index of the task is determined based on the minimum angle of the triangle. The tasks are then sorted and grouped for sampling to construct the task set to be labeled.
It increases the diversity of task sets, reduces redundancy, and improves the quality and accuracy of model training data.
Smart Images
Figure CN115774843B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of model training technology, and in particular to a method, apparatus and electronic device for constructing a triplet task set. Background Technology
[0002] Artificial intelligence technology has developed rapidly in recent years, and the demand for data from existing deep learning frameworks is increasing daily. Constructing training data has become a key bottleneck in machine learning applications. Active learning can use machine learning methods to select suitable candidate sets for human labeling, and then use the manually labeled data to train supervised or semi-supervised learning models, thereby gradually improving model performance. In practical applications, dataset labels are often obtained through outsourcing, and data crowdsourcing is a widely adopted method for obtaining labels in recent years. Crowdsourcing platforms attract online users to perform large-scale data labeling, and then aggregate the results of multiple labelers to obtain task labels. However, in general active learning strategies for specific triplet tasks on crowdsourcing platforms, simply measuring sample uncertainty using information entropy is not only lacking in specificity but also carries the possibility of filtering redundant samples, making the design of specific strategies challenging. Summary of the Invention
[0003] The purpose of this application is to provide a method, apparatus, and electronic device for constructing a triplet task set, which can sort tasks based on the difficulty evaluation index parameters of each task in the unlabeled triplet task set, perform task grouping and sampling based on the task sorting results, and construct a triplet task set to be labeled, thereby improving the task diversity in the task set while reducing redundancy.
[0004] In a first aspect, embodiments of this application provide a method for constructing a triplet task set, the method comprising: obtaining an unlabeled triplet task set; the unlabeled triplet task set including multiple unlabeled triplet tasks; determining a difficulty evaluation index parameter for each triplet task in the unlabeled triplet task set; the difficulty evaluation index parameter being used to characterize the difficulty level of the triplet task when it is labeled; sorting the triplet tasks in the unlabeled triplet task set according to the difficulty evaluation index parameter of each triplet task; and performing task grouping sampling based on the task sorting results to obtain a triplet task set to be labeled.
[0005] In a preferred embodiment of this application, the step of determining the difficulty evaluation index parameter of each triplet task in the unlabeled triplet task set includes: for each triplet task in the unlabeled triplet task set, obtaining three feature vectors with the same starting point corresponding to the triplet task; and determining the difficulty evaluation index parameter of the triplet task based on the three feature vectors.
[0006] In a preferred embodiment of this application, the step of obtaining three feature vectors with the same starting point corresponding to the triplet task includes: inputting three samples from the triplet task into a pre-trained vector mapping model to obtain three feature vectors with the same starting point.
[0007] In a preferred embodiment of this application, the step of determining the difficulty evaluation index parameters of the triplet task based on the three feature vectors includes: constructing a triangle for evaluating the difficulty of task labeling based on the three feature vectors; and determining the difficulty evaluation index parameters corresponding to the triplet task based on the triangle.
[0008] In a preferred embodiment of this application, the step of constructing a triangle for evaluating the difficulty of task annotation based on three feature vectors includes: connecting the endpoints corresponding to the three feature vectors to obtain a triangle for evaluating the difficulty of task annotation.
[0009] In a preferred embodiment of this application, the step of determining the difficulty evaluation index parameter corresponding to the triplet task based on the triangle includes: calculating the three side lengths of the triangle based on the three feature vectors; calculating the three angles corresponding to the triangle based on the three side lengths and the cosine function; determining the difficulty evaluation index parameter corresponding to the triplet task based on the minimum angle among the three angles; the minimum angle is proportional to the difficulty evaluation index parameter.
[0010] In a preferred embodiment of this application, the step of calculating the three side lengths of the triangle based on the three eigenvectors includes: obtaining the endpoint coordinates corresponding to the three eigenvectors respectively; calculating the Euclidean distance between each pair of endpoint coordinates; and using the Euclidean distance between each pair of endpoints as the three side lengths of the triangle.
[0011] In a preferred embodiment of this application, the step of determining the difficulty evaluation index parameter corresponding to the triplet task based on the smallest angle among the three angles includes: determining the smallest angle among the three angles as the difficulty evaluation index parameter corresponding to the triplet task; or, converting the smallest angle into a value between 0 and 1, and using the value as the difficulty evaluation index parameter corresponding to the triplet task.
[0012] In a preferred embodiment of this application, the step of grouping and sampling tasks based on task ranking results to obtain a set of triplet tasks to be labeled includes: dividing the tasks in the unlabeled triplet task set into multiple groups according to the difficulty evaluation index parameter; extracting a specified number of triplet tasks for each group; and constructing a set of triplet tasks to be labeled from the triplet tasks extracted from each group.
[0013] In a preferred embodiment of this application, the step of dividing the tasks in the unlabeled triplet task set into multiple groups of tasks according to the magnitude of the difficulty evaluation index parameter includes: uniformly dividing the numerical range formed by the minimum and maximum values of the difficulty evaluation index parameter into multiple sub-intervals; and treating the triplet tasks corresponding to each sub-interval as a group of tasks.
[0014] In a preferred embodiment of this application, the step of extracting a specified number of triplet tasks for each group of tasks includes: extracting a specified number of triplet tasks from each group of tasks in descending order of difficulty evaluation index parameters.
[0015] Secondly, embodiments of this application also provide an apparatus for constructing a triplet task set. The apparatus includes: a task set acquisition module, used to acquire an unlabeled triplet task set; the unlabeled triplet task set includes multiple unlabeled triplet tasks; a difficulty evaluation module, used to determine the difficulty evaluation index parameter of each triplet task in the unlabeled triplet task set; the difficulty evaluation index parameter is used to characterize the difficulty level of the triplet task when it is labeled; a sorting module, used to sort the triplet tasks in the unlabeled triplet task set according to the difficulty evaluation index parameter of each triplet task; and a group sampling module, used to construct triplet tasks based on the task sorting results to obtain a triplet task set to be labeled.
[0016] Thirdly, embodiments of this application also provide an electronic device, including a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the method described in the first aspect above.
[0017] Fourthly, embodiments of this application also provide a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are invoked and executed by a processor, the computer-executable instructions cause the processor to implement the method described in the first aspect above.
[0018] This application provides a method, apparatus, and electronic device for constructing a triplet task set. First, an unlabeled triplet task set is obtained; this unlabeled triplet task set includes multiple unlabeled triplet tasks. A difficulty evaluation index parameter is determined for each triplet task in the unlabeled triplet task set; the difficulty evaluation index parameter characterizes the difficulty level of the triplet task when it is labeled. Based on the difficulty evaluation index parameter of each triplet task, the triplet tasks in the unlabeled triplet task set are sorted. Based on the task sorting results, task grouping and sampling are performed to obtain the triplet task set to be labeled. This application embodiment, by sorting tasks based on the difficulty evaluation index parameter of each task in the unlabeled triplet task set and performing task grouping and sampling based on the task sorting results, constructs a triplet task set to be labeled that has high task diversity and low redundancy. Attached Figure Description
[0019] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0020] Figure 1 A flowchart illustrating a method for constructing a triplet task set as provided in an embodiment of this application;
[0021] Figure 2 A flowchart illustrating the method for determining difficulty evaluation index parameters in a method for constructing a triplet task set provided in this application embodiment;
[0022] Figure 3 A schematic diagram of the feature vector corresponding to a triplet task provided in an embodiment of this application;
[0023] Figure 4 A schematic diagram illustrating a difficult distribution scenario provided in an embodiment of this application;
[0024] Figure 5 A schematic diagram of a triangular Euclidean distance provided in an embodiment of this application;
[0025] Figure 6 A simplified distribution triangle diagram provided for an embodiment of this application;
[0026] Figure 7 A schematic diagram of a difficulty distribution triangle provided in this application embodiment;
[0027] Figure 8 This application provides a schematic diagram of group sampling as an embodiment of the present application.
[0028] Figure 9 A flowchart illustrating the overall task processing provided in this application embodiment;
[0029] Figure 10 A structural block diagram of a triplet task set construction apparatus provided in an embodiment of this application;
[0030] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0031] The technical solutions of this application will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0032] Taking the triplet task on a crowdsourcing platform as an example, in each round of annotation, the annotator will obtain a task package with three samples to be annotated on the annotation page. The three sample ground truth labels correspond to anchor, positive, and negative, respectively. Anchor represents the target to be compared, positive represents a target similar to the target to be compared, and negative represents a target different from the target to be compared. Due to the similarity in the distribution of anchor and positive in the feature space, the annotator only needs to select the most unique sample in the task package on the crowdsourcing platform as the negative sample. The above is the annotation approach for the triplet annotation task on the crowdsourcing platform.
[0033] In this embodiment of the application, the triplet task is defined as follows: during the triplet annotation task, the annotator only needs to select the sample with the highest similarity within the group. For example, for sample ABC, if the annotator believes that A and B have the highest similarity among the three samples, then the annotator will select C as the negative sample for this question, that is, give the unique sample a negative label.
[0034] Current active learning strategies for triplet tasks mainly tend to measure uncertainty through information entropy or evaluate sample diversity through methods such as clustering. However, since these methods are merely generalizations of the idea, there are currently no diversity strategies specifically designed for triplet tasks.
[0035] Uncertainty sampling in active learning involves treating samples whose boundaries the model judges as high-uncertainty samples. These samples are then fed into an expert system for labeling and updating the model's training, thereby improving the model's ability to distinguish difficult-to-distinguish samples. The minimum confidence strategy is often used in binary or multi-class classification models, determining the category of a sample based on its confidence score. For example, in a binary classification scenario, if two data points are predicted by a classifier, this strategy selects the sample with the lowest maximum probability value for labeling. Entropy, in mathematics, measures the uncertainty of a system; a higher entropy value indicates greater uncertainty, and a lower entropy value indicates less uncertainty. Therefore, in some classification problems, samples with higher entropy values are often selected as high-uncertainty samples to construct the labeled dataset.
[0036] Diversity sampling in active learning strategies is also a common strategy that considers the distribution of data. Diversity algorithms often ensure that the queried samples can cover the entire data through data distribution, thereby ensuring the diversity of labeled samples. The commonly used methods include the following categories: (1) Model-based outlier: constructing the dataset to be labeled using outlier samples that make the model less active; (2) Representative sampling: selecting some of the most representative samples from the unlabeled sample pool, such as obtaining representative samples by clustering or by obtaining representative samples through the differences in domain distribution; (3) Real-world scene diversity: completing sampling through the diversity and distribution of real-world scenes.
[0037] Existing active learning methods on crowdsourcing platforms primarily sample from unlabeled sample pools based on uncertainty and sample diversity to select the most valuable samples. However, selecting samples based on uncertainty generates a large number of redundant samples in batch selection scenarios, while selecting samples based on diversity introduces outliers resembling noise. Furthermore, existing evaluation metrics for active learning strategies are mainly designed for classification or regression tasks, lacking evaluation metrics specifically for triplet tasks. Therefore, the effectiveness of existing strategies cannot be reasonably validated when calculating the uncertainty or diversity of sample data, potentially leading to discrepancies in evaluations and a decline in model performance.
[0038] Based on this, embodiments of this application provide a method, apparatus, and electronic device for constructing a triplet task set, which can sort tasks based on the difficulty evaluation index parameters of each task in the unlabeled triplet task set, perform task grouping and sampling based on the task sorting results, and construct a triplet task set to be labeled, thereby improving the diversity of tasks in the task set while reducing redundancy.
[0039] To facilitate understanding of this embodiment, a method for constructing a triplet task set disclosed in this application embodiment will first be described in detail.
[0040] Figure 1 A flowchart illustrating a method for constructing a triplet task set provided in this application embodiment is shown. The method specifically includes the following steps:
[0041] Step S102: Obtain the set of unlabeled triple tasks; the set of unlabeled triple tasks includes multiple unlabeled triple tasks.
[0042] In practice, the unlabeled data under triplet tasks within the crowdsourcing platform can be used to construct an unlabeled triplet task set. That is, each task in the triplet task set is an unlabeled triplet task.
[0043] Step S104: Determine the difficulty evaluation index parameters for each triplet task in the unlabeled triplet task set.
[0044] The aforementioned difficulty evaluation index parameters are used to characterize the difficulty level of the triplet task when it is labeled. There are various ways to determine the labeling difficulty of a triplet task. For example, based on the feature vectors corresponding to the three samples in the triplet task, a triangle for evaluating the difficulty evaluation index parameters can be constructed, and then the labeling difficulty can be evaluated by determining the minimum angle within the triangle. In this embodiment, the difficulty evaluation index parameters are characterized by the minimum angle of the triangle formed by the three feature vectors corresponding to the three samples in the triplet task; or, the minimum angle can be converted to a value between 0 and 1 for characterization.
[0045] Step S106: Sort the triplet tasks in the unlabeled triplet task set according to the difficulty evaluation index parameters of each triplet task.
[0046] Tasks in the task set can be sorted in ascending order of difficulty evaluation index parameters, or they can be sorted in descending order of difficulty evaluation index parameters.
[0047] Step S108: Based on the task ranking results, perform task grouping sampling to obtain the set of triplet tasks to be labeled.
[0048] The tasks are sorted by difficulty, grouped by difficulty, and a certain number of tasks are extracted from each group to obtain the set of triplet tasks to be labeled.
[0049] This application provides a method for constructing a triplet task set. First, an unlabeled triplet task set is obtained, comprising multiple unlabeled triplet tasks. Then, the difficulty evaluation index parameter for each triplet task in the unlabeled triplet task set is determined. Based on the difficulty evaluation index parameter, the triplet tasks in the unlabeled triplet task set are ranked. Finally, task grouping and sampling are performed based on the task ranking results to obtain the triplet task set to be labeled. This application, through the above method, can rank tasks based on the difficulty evaluation index parameter of each task in the unlabeled triplet task set and perform task grouping and sampling based on the task ranking results, constructing a triplet task set to be labeled with high task diversity and low redundancy.
[0050] This application also provides a method for constructing a triplet task set, which is implemented based on the previous embodiment. This embodiment focuses on describing the specific determination method of the difficulty evaluation index parameters and the group sampling process.
[0051] See Figure 2 As shown, the process of determining the difficulty evaluation index parameters for each triplet task in the unlabeled triplet task set specifically includes the following steps:
[0052] Step S202: For each triplet task in the unlabeled triplet task set, obtain three feature vectors with the same starting point corresponding to the triplet task.
[0053] In practice, the three samples from the triplet task can be input into a pre-trained vector mapping model to obtain three feature vectors with the same starting point.
[0054] The pre-trained vector mapping model can be used as a knowledge distillation framework to achieve the goals of representation learning and triple classification. During model training, the student model is trained based on a known and fixed teacher model, specifically by requiring the representation vectors obtained by the student model to be similar to the representations output by the teacher model.
[0055] Mapping three samples in a triplet task to three feature vectors using a trained model refers to mapping the triplet data to a high-dimensional space through a student model and networks such as MLP to obtain feature vectors. Feature vectors can be considered a high-dimensional implicit encoding method.
[0056] The three samples in the triplet task are mapped to feature vectors, vec1, vec2, and vec3, using a trained model, as follows: Figure 3 As shown. The space where the feature vectors reside is a high-dimensional space of N dimensions. Figure 3 The representation in the text is simply a two-dimensional representation for ease of understanding.
[0057] Step S204: Determine the difficulty evaluation index parameters of the triplet task based on the three feature vectors.
[0058] In practice, the following methods can be used:
[0059] (1) Construct a triangle for evaluating the difficulty of task annotation based on three feature vectors. Connect the endpoints of the three feature vectors to obtain the triangle for evaluating the difficulty of task annotation.
[0060] In triplet tasks, when one feature vector significantly deviates from the other two, and these two feature vectors have highly similar distributions in the feature space, the model can easily determine that the most unique sample in the task is the original sample corresponding to the highly deviated feature vector. However, when the three feature vectors have highly similar distributions in the feature space, or when there are significant deviations between each pair of vectors, the model struggles to select the most unique sample based on the feature vector distribution in the feature space. Specific scenarios include... Figure 4 As shown.
[0061] Depend on Figure 4 As shown in the simple and hard distributions, the triangle formed by connecting the endpoints of the three feature vectors reflects the difficulty of the task annotation. For example, in the simple distribution, the minimum angle of the triangle formed by connecting the endpoints of the three feature vectors is very small, while in the hard distribution, the minimum angle of the triangle formed by connecting the endpoints of the three feature vectors is close to 60 degrees. That is to say, the difficulty of the task annotation can be determined by the size of the minimum angle of the triangle. Therefore, in this embodiment, for the triplet task, a triangle is constructed using its three corresponding feature vectors, and the task annotation difficulty is evaluated based on the minimum angle of the triangle, as follows:
[0062] (2) Determine the difficulty evaluation index parameters corresponding to the triplet task based on the triangle. The specific process for determining the difficulty evaluation index parameters is as follows:
[0063] A. Calculate the three side lengths of the triangle based on the three eigenvectors; in practice, obtain the endpoint coordinates corresponding to the three eigenvectors respectively; calculate the Euclidean distance between each pair of endpoint coordinates; use the Euclidean distance between each pair of endpoints as the three side lengths of the triangle.
[0064] In this embodiment, the goal is to express the differences between three eigenvectors using the parameters of a triangle. Therefore, it is only necessary to calculate the distance between the endpoints of each pair of eigenvectors. The distance representation in high-dimensional space is Euclidean distance, see [link to documentation]. Figure 5As shown, d1, d2, and d3 represent the Euclidean distances between vec1 and vec2, vec2 and vec3, and vec1 and vec3, respectively, which are the lengths of the three sides.
[0065] B. Calculate the three angles of the triangle based on the three side lengths and the cosine function.
[0066] The cosine theorem, corresponding to the cosine function, means that for any triangle, the square of any side is equal to the sum of the squares of the other two sides minus twice the product of those two sides and the cosine of the angle between them.
[0067] C. Determine the difficulty evaluation index parameter corresponding to the triplet task based on the smallest of the three angles; the smallest angle is directly proportional to the difficulty evaluation index parameter. That is, the smaller the smallest angle, the lower the difficulty of characterization and annotation; the larger the smallest angle, the higher the difficulty of characterization and annotation.
[0068] Specifically, the smallest of the three angles can be determined as the difficulty evaluation index parameter corresponding to the triplet task; or, the smallest angle can be converted into a value between 0 and 1, and the value can be used as the difficulty evaluation index parameter corresponding to the triplet task.
[0069] See Figure 6 As shown in the simple task triangle, due to the existence of a feature vector with a large deviation, the shape of this distance triangle exhibits a distinct "two long and one short" sharp shape, and the minimum angle value inside this triangle also tends towards zero. From... Figure 7 As shown in the difficult task triangle, when the distributions of the three feature vectors are highly similar (left) or there is a large distribution deviation between each pair (right), the constructed distance triangle presents a "three-sided approximation" shape, and the minimum angle value inside the triangle is more inclined to be 60°.
[0070] Therefore, in this embodiment, the minimum angle value of the triangle constructed from the feature vectors is used to define the difficulty evaluation index parameter of the triplet task. Since the maximum value of the interior angle of the triangle does not exceed 60°, the score of this difficulty evaluation index parameter can be set as follows:
[0071] score=(minΔ angle ) / 60, score∈[0,1];
[0072] The higher the score, the closer the distance from the maximum value of the smallest interior angle of the triangle is to 60°, and the higher the difficulty of the task set; conversely, the lower the score, the closer the distance from the maximum value of the smallest interior angle of the triangle is to 0°, and the lower the difficulty of the task set. After determining the difficulty evaluation index parameters corresponding to each triplet task in the above manner, all tasks in the unlabeled triplet task set can be sorted.
[0073] The specific task sampling process is explained in detail below:
[0074] The task ranking results mentioned above include: multiple tasks in the unlabeled triplet task set arranged in order of difficulty evaluation index parameters; the step of grouping and sampling tasks based on the task ranking results to obtain the task set to be labeled triplet is implemented in the following way:
[0075] (1) Divide the tasks in the unlabeled triplet task set into multiple task groups according to the size of the difficulty evaluation index parameter;
[0076] In practice, the numerical range formed by the minimum and maximum values of the difficulty evaluation index parameters can be evenly divided into multiple sub-intervals; the triplet tasks corresponding to each sub-interval can be regarded as a group of tasks.
[0077] After sorting the tasks in the task set according to the order of difficulty evaluation index parameters, the task set is as follows: Figure 8 As shown in (a), the left end represents the low-score samples, and the right end represents the high-score samples. Let the lowest value be min and the highest value be max. Divide this (min, max) range into n equal parts, where (min, θ1, ..., θ2) = n. n-1 The numbers (max) are distributed in an arithmetic sequence, as shown in the diagram below. Figure 8 As shown in (b) (n=4 in this example figure).
[0078] (2) For each group of tasks, select a specified number of triplet tasks; in practice, a specified number of triplet tasks can be selected from each group of tasks in descending order of difficulty evaluation index parameters.
[0079] Within each task group, samples were taken from highest to lowest difficulty level (n). topk =N topk (n) tasks, as illustrated in the diagram. Figure 8 As shown in (c).
[0080] (3) The set of triplets to be labeled is formed by the triplets extracted from each group.
[0081] Finally, group n into n topk The tasks are combined to form the set of tasks to be labeled, which was sampled based on the difficulty diversity index, as shown in the diagram. Figure 8 As shown in (d), the final set of tasks to be labeled with triples contains N tasks. top k.
[0082] See Figure 9 As shown, after constructing the set of triplet tasks to be labeled, it is sent to online labelers for labeling. After labeling, it can be used as training data for model training. The model trained based on this training data will have higher accuracy.
[0083] The method for constructing a triplet task set provided in this application determines the difficulty evaluation index parameter of the triplet task by using the feature vectors corresponding to the three samples in the triplet task. Then, all tasks in the task set are sorted based on the parameter. Task grouping and sampling are performed according to the sorting results. That is, the total tasks are divided into multiple groups according to the difficulty evaluation index parameter, and an equal number of tasks are evenly extracted from each group. Thus, a diverse task package is constructed based on the difficulty index for annotators to complete the annotation. The task set constructed in this way ensures task diversity on the one hand, and reduces the sampling frequency of redundant samples to a certain extent on the other hand.
[0084] Based on the above method embodiments, this application also provides an apparatus for constructing a triplet task set, see [link to relevant documentation]. Figure 10 As shown, the device includes: a task set acquisition module 102, used to acquire an unlabeled triplet task set; the unlabeled triplet task set includes multiple unlabeled triplet tasks; a difficulty evaluation module 104, used to determine the difficulty evaluation index parameter of each triplet task in the unlabeled triplet task set; the difficulty evaluation index parameter is used to characterize the difficulty level of the triplet task when it is labeled; a sorting module 106, used to sort the triplet tasks in the unlabeled triplet task set according to the difficulty evaluation index parameter of each triplet task; and a group sampling module 108, used to perform task group sampling based on the task sorting results to obtain a triplet task set to be labeled.
[0085] In a preferred embodiment of this application, the above-mentioned difficulty evaluation module 104 is used to take each triple task in the unlabeled triple task set as a triple task and perform the following steps: map the three samples in the triple task into three feature vectors through a pre-trained vector mapping model; the three feature vectors have the same starting point; construct a triangle for evaluating the difficulty of task labeling based on the three feature vectors; and determine the difficulty evaluation index parameters corresponding to the triple task based on the triangle.
[0086] In a preferred embodiment of this application, the aforementioned difficulty evaluation module 104 is used to connect the endpoints corresponding to the three feature vectors to obtain a triangle for evaluating the difficulty of task labeling.
[0087] In a preferred embodiment of this application, the aforementioned difficulty evaluation module 104 is used to calculate the three side lengths of a triangle based on three feature vectors; calculate the three angles corresponding to the triangle based on the three side lengths and the cosine function; determine the difficulty evaluation index parameter corresponding to the triplet task based on the minimum angle among the three angles; and the minimum angle is proportional to the difficulty evaluation index parameter.
[0088] In a preferred embodiment of this application, the aforementioned difficulty evaluation module 104 is used to obtain the endpoint coordinates corresponding to the three feature vectors respectively; calculate the Euclidean distance between each pair of endpoint coordinates; and use the Euclidean distance between each pair of endpoints as the three side lengths of the triangle.
[0089] In a preferred embodiment of this application, the aforementioned difficulty evaluation module 104 is used to determine the smallest angle among the three angles as the difficulty evaluation index parameter corresponding to the triplet task; or, to convert the smallest angle into a value between 0 and 1, and use the value as the difficulty evaluation index parameter corresponding to the triplet task.
[0090] In a preferred embodiment of this application, the group sampling module 108 is used to divide the tasks in the unlabeled triplet task set into multiple groups of tasks according to the difficulty evaluation index parameter; for each group of tasks, a specified number of triplet tasks are extracted; and the triplet tasks extracted from each group constitute the triplet task set to be labeled.
[0091] In a preferred embodiment of this application, the group sampling module 108 is used to evenly divide the numerical range formed by the minimum and maximum values of the difficulty evaluation index parameters into multiple sub-intervals; and to treat the triplet tasks corresponding to each sub-interval as a group of tasks.
[0092] In a preferred embodiment of this application, the group sampling module 108 is used to extract a specified number of triplet tasks from each group of tasks in descending order of difficulty evaluation index parameters.
[0093] The device provided in this application embodiment has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts of the device embodiment not mentioned can be referred to the corresponding content in the aforementioned method embodiment.
[0094] This application also provides an electronic device, such as... Figure 11 The diagram shows the structure of the electronic device, which includes a processor 111 and a memory 110. The memory 110 stores computer-executable instructions that can be executed by the processor 111. The processor 111 executes the computer-executable instructions to implement the following method steps:
[0095] Obtain an unlabeled triplet task set; the unlabeled triplet task set includes multiple unlabeled triplet tasks; determine the difficulty evaluation index parameter for each triplet task in the unlabeled triplet task set; the difficulty evaluation index parameter is used to characterize the difficulty level of the triplet task when it is labeled; sort the triplet tasks in the unlabeled triplet task set according to the difficulty evaluation index parameter of each triplet task; perform task grouping sampling based on the task sorting results to obtain the triplet task set to be labeled.
[0096] In a preferred embodiment of this application, the step of determining the difficulty evaluation index parameter of each triplet task in the unlabeled triplet task set includes: taking each triplet task in the unlabeled triplet task set as a triplet task, performing the following steps: mapping the three samples in the triplet task into three feature vectors through a pre-trained vector mapping model; the three feature vectors have the same starting point; constructing a triangle for evaluating the difficulty of task labeling based on the three feature vectors; and determining the difficulty evaluation index parameter corresponding to the triplet task based on the triangle.
[0097] In a preferred embodiment of this application, the step of constructing a triangle for evaluating the difficulty of task annotation based on three feature vectors includes: connecting the endpoints corresponding to the three feature vectors to obtain a triangle for evaluating the difficulty of task annotation.
[0098] In a preferred embodiment of this application, the step of determining the difficulty evaluation index parameter corresponding to the triplet task based on the triangle includes: calculating the three side lengths of the triangle based on the three feature vectors; calculating the three angles corresponding to the triangle based on the three side lengths and the cosine function; determining the difficulty evaluation index parameter corresponding to the triplet task based on the minimum angle among the three angles; the minimum angle is proportional to the difficulty evaluation index parameter.
[0099] In a preferred embodiment of this application, the step of calculating the three side lengths of the triangle based on the three eigenvectors includes: obtaining the endpoint coordinates corresponding to the three eigenvectors respectively; calculating the Euclidean distance between each pair of endpoint coordinates; and using the Euclidean distance between each pair of endpoints as the three side lengths of the triangle.
[0100] In a preferred embodiment of this application, the step of determining the difficulty evaluation index parameter corresponding to the triplet task based on the smallest angle among the three angles includes: determining the smallest angle among the three angles as the difficulty evaluation index parameter corresponding to the triplet task; or, converting the smallest angle into a value between 0 and 1, and using the value as the difficulty evaluation index parameter corresponding to the triplet task.
[0101] In a preferred embodiment of this application, the step of grouping and sampling tasks based on task ranking results to obtain a set of triplet tasks to be labeled includes: dividing the tasks in the unlabeled triplet task set into multiple groups according to the difficulty evaluation index parameter; extracting a specified number of triplet tasks for each group; and constructing a set of triplet tasks to be labeled from the triplet tasks extracted from each group.
[0102] In a preferred embodiment of this application, the step of dividing the tasks in the unlabeled triplet task set into multiple groups of tasks according to the magnitude of the difficulty evaluation index parameter includes: uniformly dividing the numerical range formed by the minimum and maximum values of the difficulty evaluation index parameter into multiple sub-intervals; and treating the triplet tasks corresponding to each sub-interval as a group of tasks.
[0103] In a preferred embodiment of this application, the step of extracting a specified number of triplet tasks for each group of tasks includes: extracting a specified number of triplet tasks from each group of tasks in descending order of difficulty evaluation index parameters.
[0104] exist Figure 11 In the illustrated embodiment, the electronic device further includes a bus 112 and a communication interface 113, wherein the processor 111, the communication interface 113, and the memory 110 are connected via the bus 112.
[0105] The memory 110 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 113 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc. The bus 112 may be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 112 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 11 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.
[0106] Processor 111 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 111 or by software instructions. The processor 111 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in the memory, and the processor 111 reads the information in the memory and, in conjunction with its hardware, completes the steps of the method described in the foregoing embodiment.
[0107] This application also provides a computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are invoked and executed by a processor, they cause the processor to perform the following method steps:
[0108] Obtain an unlabeled triplet task set; the unlabeled triplet task set includes multiple unlabeled triplet tasks; determine the difficulty evaluation index parameter for each triplet task in the unlabeled triplet task set; the difficulty evaluation index parameter is used to characterize the difficulty level of the triplet task when it is labeled; sort the triplet tasks in the unlabeled triplet task set according to the difficulty evaluation index parameter of each triplet task; perform task grouping sampling based on the task sorting results to obtain the triplet task set to be labeled.
[0109] In a preferred embodiment of this application, the step of determining the difficulty evaluation index parameter of each triplet task in the unlabeled triplet task set includes: taking each triplet task in the unlabeled triplet task set as a triplet task, performing the following steps: mapping the three samples in the triplet task into three feature vectors through a pre-trained vector mapping model; the three feature vectors have the same starting point; constructing a triangle for evaluating the difficulty of task labeling based on the three feature vectors; and determining the difficulty evaluation index parameter corresponding to the triplet task based on the triangle.
[0110] In a preferred embodiment of this application, the step of constructing a triangle for evaluating the difficulty of task annotation based on three feature vectors includes: connecting the endpoints corresponding to the three feature vectors to obtain a triangle for evaluating the difficulty of task annotation.
[0111] In a preferred embodiment of this application, the step of determining the difficulty evaluation index parameter corresponding to the triplet task based on the triangle includes: calculating the three side lengths of the triangle based on the three feature vectors; calculating the three angles corresponding to the triangle based on the three side lengths and the cosine function; determining the difficulty evaluation index parameter corresponding to the triplet task based on the minimum angle among the three angles; the minimum angle is proportional to the difficulty evaluation index parameter.
[0112] In a preferred embodiment of this application, the step of calculating the three side lengths of the triangle based on the three eigenvectors includes: obtaining the endpoint coordinates corresponding to the three eigenvectors respectively; calculating the Euclidean distance between each pair of endpoint coordinates; and using the Euclidean distance between each pair of endpoints as the three side lengths of the triangle.
[0113] In a preferred embodiment of this application, the step of determining the difficulty evaluation index parameter corresponding to the triplet task based on the smallest angle among the three angles includes: determining the smallest angle among the three angles as the difficulty evaluation index parameter corresponding to the triplet task; or, converting the smallest angle into a value between 0 and 1, and using the value as the difficulty evaluation index parameter corresponding to the triplet task.
[0114] In a preferred embodiment of this application, the step of grouping and sampling tasks based on task ranking results to obtain a set of triplet tasks to be labeled includes: dividing the tasks in the unlabeled triplet task set into multiple groups according to the difficulty evaluation index parameter; extracting a specified number of triplet tasks for each group; and constructing a set of triplet tasks to be labeled from the triplet tasks extracted from each group.
[0115] In a preferred embodiment of this application, the step of dividing the tasks in the unlabeled triplet task set into multiple groups of tasks according to the magnitude of the difficulty evaluation index parameter includes: uniformly dividing the numerical range formed by the minimum and maximum values of the difficulty evaluation index parameter into multiple sub-intervals; and treating the triplet tasks corresponding to each sub-interval as a group of tasks.
[0116] In a preferred embodiment of this application, the step of extracting a specified number of triplet tasks for each group of tasks includes: extracting a specified number of triplet tasks from each group of tasks in descending order of difficulty evaluation index parameters.
[0117] The computer program products of the methods, apparatus, and electronic devices provided in the embodiments of this application include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the preceding method embodiments. For specific implementations, please refer to the method embodiments, which will not be repeated here.
[0118] Unless otherwise specifically stated, the relative steps, numerical expressions, and values of the components and steps described in these embodiments do not limit the scope of this application.
[0119] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0120] In the description of this application, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.
[0121] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the technical scope disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.
Claims
1. A method for constructing a triplet task set, characterized in that, The method includes: Obtain an unlabeled triple task set; the unlabeled triple task set includes multiple unlabeled triple tasks, and the three sample true labels in the triple correspond to anchor, positive and negative respectively. The anchor represents the target to be compared, the positive represents the target similar to the target to be compared, and the negative represents the target different from the target to be compared. For each triplet task in the unlabeled triplet task set, obtain three feature vectors with the same starting point corresponding to the triplet task; The difficulty evaluation index parameters of the triplet task are determined based on the three feature vectors; the difficulty evaluation index parameters are used to characterize the difficulty level of the triplet task when it is labeled. Based on the difficulty evaluation index parameters of each triplet task, the triplet tasks in the unlabeled triplet task set are sorted. The numerical range formed by the minimum and maximum values of the difficulty evaluation index parameters is evenly divided into multiple sub-intervals; Treat the triplet tasks corresponding to each of the sub-intervals as a group of tasks; For each group of tasks, a specified number of triplet tasks are drawn; The set of triplet tasks to be labeled is composed of the triplet tasks extracted from each group.
2. The method according to claim 1, characterized in that, The steps for obtaining the three feature vectors with the same starting point corresponding to the triplet task include: The three samples in the triplet task are input into a pre-trained vector mapping model to obtain three feature vectors with the same starting point.
3. The method according to claim 1, characterized in that, The steps for determining the difficulty evaluation index parameters of the triplet task based on the three feature vectors include: A triangle for evaluating the difficulty of task annotation is constructed based on the three feature vectors; The difficulty evaluation index parameters corresponding to the triplet task are determined based on the triangle.
4. The method according to claim 3, characterized in that, The steps for constructing a triangle for evaluating the difficulty of task annotation based on the three feature vectors include: Connecting the endpoints of the three feature vectors yields a triangle used to evaluate the difficulty of task annotation.
5. The method according to claim 3, characterized in that, The steps for determining the difficulty evaluation index parameters corresponding to the triplet task based on the triangle include: Calculate the lengths of the three sides of the triangle based on the three feature vectors; Based on the three side lengths and the cosine function, calculate the three angles corresponding to the triangle; The difficulty evaluation index parameter corresponding to the triplet task is determined based on the smallest angle among the three angles; the smallest angle is proportional to the difficulty evaluation index parameter.
6. The method according to claim 5, characterized in that, The steps for calculating the three side lengths of the triangle based on the three eigenvectors include: Obtain the endpoint coordinates corresponding to the three feature vectors respectively; Calculate the Euclidean distance between the coordinates of each pair of endpoints; The Euclidean distance between each pair of endpoints is used as the three side lengths of the triangle.
7. The method according to claim 5, characterized in that, The steps for determining the difficulty evaluation index parameters corresponding to the triplet task based on the smallest of the three angles include: The smallest of the three angles is determined as the difficulty evaluation index parameter corresponding to the triplet task; Alternatively, the minimum angle can be converted into a value between 0 and 1, and the value can be used as the difficulty evaluation index parameter corresponding to the triplet task.
8. The method according to claim 1, characterized in that, For each set of tasks, the steps to extract a specified number of triplet tasks include: From each group of tasks, select a specified number of triplet tasks in descending order of difficulty evaluation index parameters.
9. A device for constructing a triplet task set, characterized in that, The device includes: The task set acquisition module is used to acquire an unlabeled triple task set; the unlabeled triple task set includes multiple unlabeled triple tasks, and the three sample true labels in the triple correspond to anchor, positive and negative respectively. The anchor represents the target to be compared, the positive represents the target similar to the target to be compared, and the negative represents the target different from the target to be compared. The difficulty evaluation module is used to obtain three feature vectors with the same starting point for each triple task in the unlabeled triple task set; determine the difficulty evaluation index parameters of the triple task based on the three feature vectors; the difficulty evaluation index parameters are used to characterize the difficulty level of the triple task when it is labeled. The sorting module is used to sort the triplet tasks in the set of unlabeled triplet tasks according to the difficulty evaluation index parameters of each triplet task. The group sampling module is used to evenly divide the numerical range formed by the minimum and maximum values of the difficulty evaluation index parameters into multiple sub-intervals. Treat the triplet tasks corresponding to each of the sub-intervals as a group of tasks; For each group of tasks, a specified number of triplet tasks are drawn; The set of triplet tasks to be labeled is composed of the triplet tasks extracted from each group.
10. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing computer-executable instructions executable by the processor, the processor executing the computer-executable instructions to implement the method of any one of claims 1 to 8.
11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when invoked and executed by a processor, cause the processor to perform the method according to any one of claims 1 to 8.