Model training method and device, equipment and storage medium

Abstract

The embodiment of the invention relates to a model training method and device, equipment and a storage medium. The method provided by the invention comprises the following steps: acquiring a plurality of original labels corresponding to a training sample, wherein the original labels indicate whether a plurality of instances exist in the training sample or not; based on the multiple original labels, label transfer information of at least one pair of instances is determined, and the label transfer information indicates that a truth value label value of a first instance in the training sample is a first value, and a truth value label value of a second instance is a third value; the possibility that the original label corresponding to the training sample mistakenly marks a second instance in the training sample as a second value; determining respective credibility of the plurality of original tags at least based on the tag transfer information corresponding to each pair of instances; and training a first machine learning model based on the training sample, the plurality of original labels and the respective credibility of the plurality of original labels, the first machine learning model being configured to detect whether a plurality of instances exist in the model input.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computer technology, and more particularly, to a model training method, apparatus, device, and storage medium. Background Art

[0002] In traditional label-based model training schemes, a training sample (such as an image sample) may be associated with multiple labels, and each label indicates whether multiple instances exist in the training sample (such as whether a cat / dog exists in the image sample). A machine learning model trained by such a model training scheme can process model inputs with complex connotations. Summary of the Invention

[0003] In a first aspect of the present disclosure, a model training method is provided. The method includes: obtaining a plurality of original labels corresponding to a plurality of training samples, where each training sample corresponds to at least two original labels, and each original label indicates whether multiple instances exist in the corresponding training sample; determining, based on the plurality of original labels corresponding to the plurality of training samples, label transition information corresponding to at least one pair of instances among the plurality of instances, where each pair of instances includes a first instance and a second instance, and the label transition information corresponding to each pair of instances indicates: when the true label value of the first instance in the training sample is a first value and the true label value of the second instance is a third value, the possibility that the original label corresponding to the training sample mislabels the second instance in the training sample as a second value; determining the credibility of each of the plurality of original labels at least based on the label transition information corresponding to each pair of instances among the plurality of instances; and training a first machine learning model based on the plurality of training samples, the plurality of original labels, and the credibility of each of the plurality of original labels, where the first machine learning model is configured to detect whether multiple instances exist in a model input.

[0004] In a second aspect of the present disclosure, there is provided an apparatus for model training. The apparatus includes: a label acquisition module configured to acquire a plurality of original labels corresponding to a plurality of training samples, where each training sample corresponds to at least two original labels, and each original label indicates whether there are multiple instances in the corresponding training sample; a transfer information determination module configured to determine label transfer information corresponding to at least a pair of instances among the multiple instances based on the plurality of original labels corresponding to the plurality of training samples, where each pair of instances includes a first instance and a second instance, and the label transfer information corresponding to each pair of instances indicates: when the true label value of the first instance in the training sample is a first value and the true label value of the second instance is a third value, the possibility that the original label corresponding to the training sample mislabels the second instance in the training sample as a second value; a credibility determination module configured to determine the credibility of each of the plurality of original labels at least based on the label transfer information corresponding to each pair of instances among the multiple instances; and a model training module configured to train a first machine learning model based on the plurality of training samples, the plurality of original labels, and the credibility of each of the plurality of original labels, where the first machine learning model is configured to detect whether there are multiple instances in the model input.

[0005] In a third aspect of the present disclosure, there is provided an electronic device. The device includes at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor. The instructions, when executed by the at least one processor, cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of the present disclosure, there is provided a computer-readable storage medium. Computer-executable instructions are stored on the computer-readable storage medium and can be executed by a processor to implement the method of the first aspect.

[0007] In a fifth aspect of the present disclosure, there is provided a computer program product. The computer program product includes computer-executable instructions that, when executed by a processor, implement the method according to the first aspect of the present disclosure.

[0008] It should be understood that the content described in this part is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. Description of the Drawings

[0009] In combination with the drawings and with reference to the following detailed description, the above and other features, advantages, and aspects of the embodiments of the present disclosure will become more apparent. In the drawings, the same or similar reference numerals denote the same or similar elements, where:

[0010] Figure 1 A schematic diagram of an environment in which embodiments of the present disclosure can be implemented is shown;

[0011] Figure 2 A flowchart showing an example process of a model training method according to some embodiments of the present disclosure;

[0012] Figure 3 A schematic structural block diagram of an apparatus for model training according to some embodiments of the present disclosure; and

[0013] Figure 4 A block diagram of an electronic device in which one or more embodiments of the present disclosure can be implemented is shown. Detailed implementation manners

[0014] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. On the contrary, these embodiments are provided to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not used to limit the protection scope of the present disclosure.

[0015] It should be noted that the titles of any sections / subsections provided herein are not restrictive. Various embodiments are described throughout this document, and any type of embodiment can be included under any section / subsection. In addition, the embodiments described in any section / subsection can be combined with any other embodiments described in the same section / subsection and / or different sections / subsections in any manner.

[0016] In the description of the embodiments of the present disclosure, the term "including" and its like should be understood as an open inclusion, that is, "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". There may also be other explicit and implicit definitions hereinafter. The terms "first", "second", etc. may refer to different or the same objects. There may also be other explicit and implicit definitions hereinafter.

[0017] Embodiments of the present disclosure may involve user data, data acquisition, and / or data usage, etc. All these aspects comply with the corresponding laws, regulations, and relevant provisions. In the embodiments of the present disclosure, the collection, acquisition, processing, processing, forwarding, usage, etc. of all data are carried out on the premise that the user is aware and has confirmed. Correspondingly, when implementing the embodiments of the present disclosure, the types, usage scopes, usage scenarios, etc. of the data or information that may be involved should be informed to the user and the user's authorization should be obtained through appropriate means in accordance with relevant laws and regulations. The specific informing and / or authorization methods may vary according to the actual situation and application scenarios, and the scope of the present disclosure is not limited in this regard.

[0018] In the solutions described in this specification and embodiments, if personal information processing is involved, it will be processed on the premise of having a legal basis (such as obtaining the consent of the personal information subject, or being necessary for performing a contract, etc.), and will only be processed within the specified or agreed scope. If the user refuses to process personal information other than the necessary information required for the basic functions, it will not affect the user's use of the basic functions.

[0019] As briefly described above, in traditional tag-based model training solutions, a training sample (such as an image sample) is associated with multiple tags, and each tag indicates whether the sample contains multiple instances. Such problems may occur in all aspects of production and life, such as text classification, acoustic classification, and gene function prediction. Traditional tag-based model training solutions assume that the samples are all correctly labeled. Unfortunately, in many real-world applications, it is difficult to obtain the correct tags.

[0020] To obtain a large number of tags required for model training, currently, the annotation data can be collected through crowdsourcing. Briefly speaking, the crowdsourcing method means jointly annotating a training sample by multiple annotators. This way of collecting tags is fast and flexible, and to a certain extent, it solves the need for a large number of tags in model training. However, due to the differences in professionalism and experience among annotators, the quality of the tags they provide may vary significantly. A large number of tags means that not every tag is reliable, and there may be annotation errors or omissions. What the learner can see are the potentially incorrect weak tags provided by multiple crowdsourcing annotators. Directly using these tags for model training will result in low model training efficiency and quality, and thus lead to biases or misjudgments in the recognition or classification tasks of the trained machine learning model.

[0021] Some current model training solutions can randomly select one tag from the crowdsourcing tags for each training sample and regard it as the correct tag. However, such model training solutions rely on a large number of correctly labeled samples to achieve good performance, which requires high annotation costs.

[0022] In order to be able to train a high-performance model with a relatively low annotation cost, some solutions provide weakly supervised model training solutions. These model training solutions only need to annotate key data to obtain a model with good performance. In addition, there are also some model training solutions that propose to utilize unannotated data by using multimodal information and label correlation. However, these solutions cannot fully utilize the potentially incorrect crowdsourced labels and cannot handle the impact brought by annotation errors, resulting in performance degradation.

[0023] In view of this, embodiments of the present disclosure provide a model training solution. According to this solution, first, a plurality of original labels corresponding to a plurality of training samples are obtained, where each training sample corresponds to at least two original labels, and each original label indicates whether there are multiple instances in the corresponding training sample. Then, based on the plurality of original labels corresponding to the plurality of training samples, label transfer information corresponding to at least one pair of instances among the plurality of instances is determined, where each pair of instances includes a first instance and a second instance, and the label transfer information corresponding to each pair of instances indicates: when the true label value of the first instance in the training sample is the first value and the true label value of the second instance is the third value, the probability that the original label corresponding to the training sample mislabels the second instance in the training sample as the second value. Next, at least based on the label transfer information corresponding to each pair of instances among the plurality of instances, the credibility of each of the plurality of original labels is determined. Subsequently, based on the plurality of training samples, the plurality of original labels, and the credibility of each of the plurality of original labels, a first machine learning model is trained, where the first machine learning model is configured to detect whether there are multiple instances in the model input.

[0024] As will be more clearly understood from the following description, in the solution of the present disclosure, the label transfer information reveals the potential transfer law of the annotation content between different labels, thereby reflecting the possibility of cross-label annotation errors between these original labels. The solution of the present disclosure further evaluates the credibility of each original label based on the label transfer information, so as to distinguish which original labels are more reliable and which original labels may have problems. During the process of model training, by introducing the credibility of each original label, multiple original labels that may contain incorrect content can be fully utilized. The machine learning model to be trained (such as the first machine learning model) can not only learn the corresponding relationship between the correct label and the instance, but also learn the impact that the annotation error may bring. For example, when the machine learning model encounters an original label with a low credibility, it can handle the information represented by these original labels more cautiously and avoid being misled by incorrect original labels.

[0025] In this way, the solution of the present disclosure can effectively identify cross-label annotation errors caused by the similarity of different category instances in the training samples by introducing label transfer information and a label credibility evaluation mechanism. The label transfer information can quickly calculate an aggregated version of the unbiased loss to mitigate the impact of cross-label annotation errors on the model performance and obtain a model with better classification effect. The solution proposed by the present disclosure effectively solves the label quality problem existing in collecting labeled data through crowdsourcing, thereby improving the effect and accuracy of model training.

[0026] Various example implementations of the solution will be described in further detail below with reference to the accompanying drawings.

[0027] Figure 1 FIG. shows a schematic diagram of an environment 100 in which embodiments of the present disclosure can be implemented. In Figure 1 the environment 100, two different stages of model training are shown, namely a training stage 101 and an application stage 102. It should be noted that although Figure 1 not shown in, there may be other stages such as a fine-tuning stage and a testing stage between the training stage 101 and the application stage 102.

[0028] In the training stage 101, the model training system 110 is configured to use the training data set 111 to perform training of the machine learning model 103. At the start of training, the machine learning model 103 has initial model parameter values. The training process of the machine learning model 103 is to update the parameter values of the model parameter values to the expected values based on the training data set 111.

[0029] In the application stage 102, the obtained machine learning model 103 has trained model parameter values and can thus be provided to the model application system 120 for use. In the application stage 102, the model application system 120 can use the machine learning model 103 to process the model input 121 in the actual scenario and provide the corresponding model output 122.

[0030] In Figure 1 the model training system 110 and the model application system 120 can include any computing system with computing capabilities, such as various computing devices / systems, terminal devices, servers, etc. The terminal device can refer to any type of mobile terminal, fixed terminal or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. The server includes but is not limited to mainframes, edge computing nodes, computing devices in cloud environments, and the like.

[0031] It should be understood that Figure 1The components and arrangements in the illustrated environment 100 are merely examples, and a computing system suitable for implementing the exemplary implementations described in this disclosure may include one or more different components, other components, and / or different arrangements. For example, although shown as separate, the model training system 110 and the model application system 120 may be integrated in the same system or device. Implementations of this disclosure are not limited in this regard.

[0032] It should be understood that the structures and functions of the various elements in the environment 100 are described only for exemplary purposes and do not imply any limitation on the scope of this disclosure.

[0033] Figure 2 A flowchart of an example process 200 of a model training method according to some embodiments of this disclosure is shown. The process 200 may be implemented at the model training system 110.

[0034] Referring to Figure 2 , at block 210, the model training system 110 obtains a plurality of original labels corresponding to a plurality of training samples. Each training sample corresponds to at least two original labels, and each original label indicates whether there are multiple instances in the corresponding training sample.

[0035] Training samples are the basic data for machine learning. Training samples can be various types of data, such as images, texts, audios, etc. For example, in an image classification task, a training sample can be an image containing a certain object, and different training samples can be images containing different objects, and so on.

[0036] Instances in a training sample can be objects to be recognized or classified in the training sample. Taking an image scenario as an example, an image sample may contain multiple objects (such as cats / dogs, etc.), and each object corresponds to an instance. Multiple instances can be categories of different objects to be recognized in the training sample. It should be understood that the embodiments of this disclosure can be applied to any other modality of samples, such as speech samples (recognizing different sound sources in speech), video samples (recognizing postures, actions, etc. in videos), text samples (recognizing different text units), and so on. Hereinafter, for the convenience of explanation, mainly the image scenario is taken as an example for illustration.

[0037] The original label can be a label provided by the annotator to indicate whether a specific instance exists in a certain training sample. The original label can be represented in any appropriate form. For example, the original label can represent in binary form whether each instance exists in the corresponding sample (such as "1, indicating that a certain instance exists" or "0, indicating that a certain instance does not exist"). For example, in an image classification task, the original label can indicate whether specific instances such as cats / dogs exist in the image. In the embodiments of the present disclosure, each training sample has two or more original labels, and these original labels can be provided by multiple annotators through crowdsourcing.

[0038] In some embodiments, different original labels corresponding to the same training sample are annotated by different annotators. Such a label annotation method can also be called a crowdsourcing annotation method. Crowdsourcing annotation relies on multiple annotators to jointly annotate a training sample. Since there are many annotators and they may come from different backgrounds and professional fields, the sample can be interpreted and annotated from multiple perspectives.

[0039] In the model training stage, after obtaining multiple training samples and their corresponding original labels, the model training system 110 will use this data to train a machine learning model (such as the machine learning model 103). The machine learning model 103 is configured to perform instance recognition or instance classification on the model input to determine whether the model input includes multiple instances. During the training process, by continuously optimizing the model parameters during the training process, the machine learning model 103 will learn the correspondence between the original label and the instance. Once the training of the machine learning model 103 is completed, it can be used for tasks such as recognition or classification. For example, in the model application stage, the user can submit an image whose content is to be recognized, and the machine learning model 103 can determine whether a specific instance exists in the image (such as whether there are cats / dogs in the image) based on the learned knowledge. It should be noted that the description of the application scenario of the machine learning model 103 here is only an exemplary description, which does not constitute a limitation on the embodiments of the present disclosure. For example, the trained machine learning model 103 can also be used for audio recognition or text classification, etc.

[0040] In block 220, the model training system 110 determines label transfer information corresponding to at least one pair of instances among multiple instances based on multiple original labels corresponding to multiple training samples. Each pair of instances includes a first instance and a second instance, and the label transfer information corresponding to each pair of instances indicates: when the true label value of the first instance in the training sample is the first value and the true label value of the second instance is the third value, the probability that the original label corresponding to the training sample mislabels the second instance in the training sample as the second value.

[0041] As an example, a pair of instances can refer to any two instances among multiple instances. The combination manner of a pair of instances is not limited to a specific rule or strategy, and it can be combined in any form according to specific application scenarios and requirements. As an example, for multiple instances c ∈ {1, ……, C}, each pair of instances can be represented as p, q ∈ {1, ……, C}, where C is a positive integer, p represents the first instance, q represents the second instance, and p and q can be the same or different.

[0042] Here, the true label value of a specific instance in the training sample can indicate the true class or attribute of the instance (although this true label value is unknown throughout the learning process). The first value can be a possible value of the true label value of instance p. As an example, in a scenario of identifying whether a certain instance exists, the first value can be represented as i ∈ {0, 1}, where i represents the first value, and a value of 0 indicates that the corresponding instance actually "does not exist" in the training sample, and a value of 1 indicates that the corresponding instance actually "exists" in the training sample. The third value can be a possible value of the true label value of instance q. As an example, in a scenario of identifying whether a certain instance exists, the third value can be represented as 1 - j ∈ {0, 1}, where 1 - j represents the third value, and a value of 0 indicates that the corresponding instance actually "does not exist" in the training sample, and a value of 1 indicates that the corresponding instance actually "exists" in the training sample. Accordingly, the second value can be represented as j ∈ {0, 1}, where j represents the second value, and a value of 0 indicates that the corresponding instance actually "does not exist" in the training sample, and a value of 1 indicates that the corresponding instance actually "exists" in the training sample. In the embodiments of the present disclosure, the first value and the second value can be equal or not equal. As an example, the first value and the second value can be represented as i, j ∈ {0, 1}.

[0043] As an example, for each pair of instances p, q, corresponding label transfer information can be determined. The label transfer information can be determined as such a transfer probability, that is, in the case where the true label value corresponding to one of the two instances p, q (for example, instance p) is determined to be a specified value (for example, i ∈ {0, 1}), the value possibility of each possible value (for example, j ∈ {0, 1}) of the original label on the other instance (for example, instance q). Thus, the label transfer information can reflect the potential influence of these two instances in terms of label annotation.

[0044] Taking the training samples as images as an example, assume that instance p corresponds to "cat", and instance q corresponds to "dog". The label transfer information can indicate the likelihood that the original label mislabels "dog" as the second value (e.g., j ∈ {0, 1}, where 1 indicates the presence of "dog" and 0 indicates the absence of "dog") when the true label value of "cat" is the first value (e.g., i ∈ {0, 1}, where 1 indicates the presence of "cat" and 0 indicates the absence of "cat"), and the true label value of "dog" is the third value (e.g., 1 - j ∈ {0, 1}, where 1 indicates the presence of "dog" and 0 indicates the absence of "dog").

[0045] It should be noted that the above descriptions about instances and label values are only exemplary. According to actual needs, instances and label values can also be represented in other forms, and the embodiments of the present disclosure do not limit this. For the convenience of discussion, unless otherwise specified, the following will take the first instance and the second instance as p and q respectively, and the first value and the second value as i and j respectively, to illustrate the embodiments of the present disclosure.

[0046] The label transfer information can be represented in any form. For example, in some embodiments, the label transfer information is represented by a transition matrix. For example, the label transfer information corresponding to instances p and q can be represented as a transition matrix T (p ,q) , where the transition matrix T (p,q) each element in can represent the likelihood that the original label labels instance q in the training sample as j when the true label value of instance p in the training sample is i (the value of the element can be represented by probability, for example). The transition matrix can intuitively display the transition probability between the label values of instances p and q, so that complex relationships can become clear and easy to calculate. Such label transfer information can reflect the likelihood of cross-label annotation errors caused by the similarity of different category instances.

[0047] It should be noted that in addition to the transition matrix, the label transfer information can also be represented in other forms to meet different scenarios and requirements. For example, the label transfer information can also be represented by probability distributions, relational graphs, or feature vectors, etc. For the convenience of discussion, unless otherwise specified, the following will illustrate the embodiments of the present disclosure by taking the label transfer information represented by the transition matrix T (p,q) as an example.

[0048] In some embodiments, the model training system 110 determines a plurality of reference samples from a training sample set based on the original labels respectively corresponding to the plurality of training samples. Then, the model training system 110 determines label transition information corresponding to at least one pair of instances among the plurality of instances based on the original labels corresponding to the plurality of reference samples. In the embodiments of the present disclosure, each reference sample is determined as an anchor for the presence or absence of a single instance among the plurality of instances (also referred to as a single-class anchor sample), or each reference sample is determined as an anchor for the presence or absence of a pair of instances among the plurality of instances (also referred to as a two-class anchor sample).

[0049] As an example, the plurality of reference samples may at least include a first training sample and a second training sample. The first training sample may be a training sample with a relatively high and / or relatively low probability of the presence of a single instance p / q (also referred to as a single-class instance p / q), and such a training sample may be determined by the model training system 110 through probability prediction with the aid of a machine learning model (such as a second machine learning model). Such a first training sample may also be referred to as a single-class anchor sample. The second training sample may be a training sample with a relatively high and / or relatively low probability of the presence of a pair of instances p, q (also referred to as a two-class instance p, q), and such a training sample may be determined by the model training system 110 through probability prediction with the aid of a machine learning model (such as a second machine learning model). Such a second training sample may also be referred to as a multi-class anchor sample (for example, specifically referred to as a two-class anchor sample).

[0050] In the embodiments of the present disclosure, a single-class anchor sample, a two-class anchor sample, or a combination of both may be selected according to the requirements of a specific task and data characteristics to determine label transition information (such as the transition matrix T (p,q) ), thereby improving the accuracy and adaptability of determining label transition information.

[0051] In some embodiments, the model training system 110 trains a second machine learning model based on the original labels respectively corresponding to the plurality of training samples. Then, the model training system 110 uses the trained second machine learning model to determine the predicted labels respectively corresponding to the plurality of training samples. Subsequently, the model training system 110 determines a plurality of reference samples from the training sample set based on the predicted labels respectively corresponding to the plurality of training samples. In the embodiments of the present disclosure, the predicted label indicates whether a plurality of instances are present in the corresponding training sample.

[0052] As an example, the second machine learning model can be any suitable machine learning model. For example, the second machine learning model can be any type of supervised (or unsupervised) learning model, such as a classifier, a regressor, etc., depending on the requirements of the task. During the training process of the second machine learning model, the second machine learning model will learn how to predict the corresponding prediction label based on the features of the training samples. This can be achieved by continuously iteratively optimizing the model parameters of the second machine learning model to minimize the difference between the prediction label and the original label. Once the second machine learning model is trained, the model training system 110 can use the second machine learning model to determine the prediction labels corresponding to the multiple training samples respectively. The prediction label can be the prediction result made by the second machine learning model according to the features of the training sample. Similar to the original label, it indicates whether there are multiple instances in the corresponding training sample (for example, whether there are cats / dogs, etc. in the training sample).

[0053] In some embodiments, the model training system 110 generates the aggregated labels corresponding to the multiple training samples respectively by performing an aggregation operation on the original labels corresponding to the multiple training samples. Then, the model training system 110 uses the multiple training samples and the aggregated labels corresponding to the multiple training samples respectively to train the second machine learning model.

[0054] As an example, the aggregation operation can be the merging, statistics or other forms of processing of the original labels to generate new, more generally meaningful aggregated labels. As an example, for each sample Aggregated label The determination process can be represented by formula (1):

[0055]

[0056] where m represents the index of the annotator (from 1 to M), M represents the total number of annotators, represents the original label provided by the m-th annotator, and the aggregated label can be a C-dimensional vector corresponding one-to-one to multiple instances. As described above, C can be the total number of multiple instances. Each dimension of the aggregated label takes a value of "0" or "1", indicating that the corresponding instance "does not exist" or "exists".

[0057] As an example, the training process of the second machine learning model can be represented by formula (2):

[0058]

[0059] where D represents the training dataset 111 containing multiple training samples x, B represents the number of samples x in the training dataset 111, Denote the loss function for the training of the second machine learning model (such as the binary cross - entropy loss function, etc.), and f(x) denote the algorithm executed by the second machine learning model.

[0060] It should be noted that the above algorithms and parameters regarding the aggregated tags and the second machine learning model are only exemplary descriptions, which do not constitute a limitation to the embodiments of the present disclosure. According to actual needs, the embodiments of the present disclosure can adopt any appropriate algorithms to determine the aggregated tags and train the second machine learning model.

[0061] In some embodiments, the model training system 110 determines, based on the prediction tags corresponding to multiple training samples, multiple first training samples corresponding to each instance among multiple instances from the multiple training samples (such as the single - class anchor samples described above). In some embodiments, the first training samples corresponding to a given instance among the multiple instances include: training samples for which the predicted value of the corresponding prediction tag for the given instance is greater than a first threshold. Alternatively or additionally, the first training samples corresponding to a given instance among the multiple instances include: training samples for which the predicted value of the corresponding prediction tag for the given instance is less than a second threshold, where the second threshold is less than the first threshold.

[0062] As an example, the model training system 110 can, based on the prediction tags, find, from the multiple training samples, several training samples for which the prediction tags indicate a relatively high and / or relatively low probability of the existence of a single instance p / q (also referred to as a single - class instance p / q), so as to obtain multiple first training samples. As an example, the determination process of the first training samples can be represented by formula (3):

[0063]

[0064] where argmax x∈D [I[i = 1]f p (x)+I[i = 0](1 - f p (x))] represents finding, from the training dataset D, the training sample x for which [I[i = 1]f p (x)+I[i = 0](1 - f p (x))] is the largest. I[·] is the Indicator Function, which is used to select an expression according to a condition. Specifically, I[i = 1] means: when i = 1, I[i = 1]=1; otherwise, I[i = 1]=0; I[i = 0] means: when i = 0, I[i = 0]=1; otherwise, I[i = 0]=0. f p (x) represents the predicted value of the prediction tag corresponding to the training sample x on the example p. Based on formula (3), the first training samples It can be understood as the training sample with the highest probability that the predicted label on instance p is i (i ∈ {0, 1}) among multiple training samples.

[0065] In an embodiment of the present disclosure, the first threshold and the second threshold can be determined according to the demand of the first training sample As an example, the model training system 110 can select, from multiple training samples, k training samples with the highest probability that the predicted label on instance p is i (i = 1) (for example, the image is most likely to contain "cat") based on the magnitude relationship between [I[i = 1]f p (x) + I[i = 0](1 - f p (x))] and the first threshold; and select, from multiple training samples, k training samples with the highest probability that the predicted label on instance p is i (i = 0) (for example, the image is least likely to contain "cat") based on the magnitude relationship between [I[i = 1]f p (x) + I[i = 0](1 - f p (x))] and the second threshold, so as to obtain a total of 2k first training samples

[0066] In some embodiments, for each first instance and second instance in at least one pair of instances, in response to determining that the first instance and the second instance indicate the same instance, the model training system 110 determines the label transfer information corresponding to the pair of instances based on the possibility (for example, which can be represented in the form of probability) that the original label corresponding to the first training sample mislabels the second instance as the second value when the true label value of the first instance is the first value.

[0067] As an example, the label transfer information corresponding to instances p and q can be represented as a transition matrix T (p,q) , where each element (p,q) in the transition matrix T can represent the possibility that the original label labels instance q in the training sample as j when the true label value of instance p in the training sample is i (the values of each element in the transition matrix can be represented by probability, for example). When the first instance and the second instance indicate the same instance (for example, p = q), the transition matrix T (p,q) can be represented by the transition matrix T (q,q) , and each element (q,q) in the transition matrix T can be determined by estimating the probability that the original label corresponding to the first training sample labels instance q as the second value j (j ∈ {0, 1}) when the true label value of instance q is the first value i (i ∈ {0, 1}).

[0068] As an example, when p = q, each element in the transition matrix T (q,q) can be estimated by Equation (4):

[0069]

[0070] where k is the number of first training samples (i.e., single-class anchor samples), denotes the original label provided by the m-th annotator when the true label value of instance q is i for the label value on instance q In this way, for each first training sample the model training system 110 can count the corresponding label values and determine the above probability information for each first training sample based on the ratio of this sum to the annotators. Then, by calculating the average value of the probability information of all first training samples the transition matrix T corresponding to a single instance q can be determined (q,q) (q,q) Thus, the transition matrix T can be quickly determined with less computational effort (q,q) (q,q)

[0071] In some embodiments, the electronic device determines, from multiple training samples, multiple second training samples corresponding to each pair of instances in at least one pair of instances based on the prediction labels respectively corresponding to the multiple training samples. In some embodiments, the multiple second training samples corresponding to each pair of instances in at least one pair of instances include: for the first instance and the second instance in each pair of instances, the training samples for which the prediction value of the corresponding prediction label for the first instance is greater than a first threshold and the prediction value for the second instance is also greater than the first threshold. Alternatively or additionally, the multiple second training samples corresponding to each pair of instances in at least one pair of instances include: for the first instance and the second instance in each pair of instances, the training samples for which the prediction value of the corresponding prediction label for the first instance is greater than a first threshold and the prediction value for the second instance is less than a second threshold, where the second threshold is less than the first threshold. Alternatively or additionally, the multiple second training samples corresponding to each pair of instances in at least one pair of instances include: for the first instance and the second instance in each pair of instances, the training samples for which the prediction value of the corresponding prediction label for the first instance is less than a second threshold and the prediction value for the second instance is greater than the first threshold. Alternatively or additionally, the multiple second training samples corresponding to each pair of instances in at least one pair of instances include: for the first instance and the second instance in each pair of instances, the training samples for which the prediction value of the corresponding prediction label for the first instance is less than a second threshold and the prediction value for the second instance is also less than the second threshold.

[0072] As an example, based on the prediction labels, the model training system 110 can find, from multiple training samples, several training samples with relatively high and / or relatively low probabilities indicating the existence of a pair of instances p and q (which can also be referred to as two types of instances p and q) in the prediction labels, so as to obtain multiple second training samples. As an example, the determination process of the second training samples can be expressed by formula (5):

[0073]

[0074] where argmax x∈D [I[i = 1]f p (x) + I[i = 0](1 - f p (x)) + I[j = 1]f q (x) + I[j = 0](1 - f q (x))] represents finding, from the training data set 111D, the training sample x that maximizes [I[i = 1]f p (x) + I[i = 0](1 - f p (x)) + I[j = 1]f q (x) + Ij = 0(1 - fqx). I[·] is an Indicator Function, which is used to select an expression according to a condition. Specifically, I[i = 1] means that when i = 1, I[i = 1] = 1; otherwise, I[i = 1] = 0; I[i = 0] means that when i = 0, I[i = 0] = 1; otherwise, I[i = 0] = 0. I[j = 1] means that when j = 1, I[j = 1] = 1; otherwise, I[j = 1] = 0; I[j = 0] means that when j = 0, I[j = 0] = 1; otherwise, I[j = 0] = 0. f p (x) represents the predicted value of the prediction label corresponding to the training sample x on the example p. Based on formula (5), the second training samples can be understood as the training samples with the highest probability that the prediction label is i (i ∈ {0, 1}) on the instance p and j (j ∈ {0, 1}) on the instance q among multiple training samples.

[0075] As described above, the first threshold and the second threshold can be determined according to the demand for the second training samples As an example, the model training system 110 can be based on [I[i = 1]f p (x) + I[i = 0](1 - f p (x)) + I[j = 1]f q (x) + I[j = 0](1 - f q(x))] and the relationship with the first threshold, from multiple training samples, select k training samples with the highest probability that the predicted label is i (i = 1) on instance p and j (j = 1) on instance q (for example, the image is most likely to contain "cat" and also contain "dog"); in addition, the model training system 110 can be based on [I[i = 1]f p (x)+I[i = 0](1 - f p (x))+I[j = 1]f q (x)+I[j = 0](1 - f q (x))] and the relationship with the first threshold and the second threshold, from multiple training samples, select k training samples with the highest probability that the predicted label is i (i = 1) on instance p and j (j = 0) on instance q (for example, the image is most likely to contain "cat" and least likely to contain "dog"), and k training samples with the highest probability that the predicted label is i (i = 0) on instance p and j (j = 1) on instance q (for example, the image is least likely to contain "cat" and most likely to contain "dog"). In addition, the model training system 110 can be based on [I[i = 1]f p (x)+I[i = 0](1 - f p (x))+I[j = 1]f q (x)+I[j = 0](1 - f q (x))] and the relationship with the second threshold, from multiple training samples, select k training samples with the highest probability that the predicted label is i (i = 0) on instance p and j (j = 0) on instance q (for example, the image is least likely to contain "cat" and also least likely to contain "dog"). Thus, the model training system 110 can obtain a total of 4k second training samples

[0076] In some embodiments, in response to determining that the first instance and the second instance indicate different instances, the model training system 110 determines the label transfer information corresponding to the pair of instances based on the likelihood (for example, it can be represented in the form of probability) that the original label corresponding to the second training sample labels the second instance as the second value when the true label value of the first instance is the first value and the true label value of the second instance is the third value.

[0077] As an example, the label transfer information corresponding to instances p and q can be represented as a transition matrix T (p,q) , where the transition matrix T (p,q) each element in It can represent the probability that, when the true label value of instance p in the training sample is i, the original label labels instance q in the training sample as j (the value of the element in the transition matrix can be represented by probability, for example). When the first instance and the second instance indicate different instances (for example, p≠q), each element in the transition matrix T (p,q) can be estimated by estimating the probability that, when the true label value of instance q is the first value i (i∈{0, 1}) and the true label value of instance q is the third value of 1-j (j∈{0, 1}), the original label corresponding to the second training sample labels instance q as the second value j (j∈{0, 1}).

[0078] As an example, when p≠q, each element in the transition matrix T (p,q) can be represented by formula (6):

[0079]

[0080] where k is the number of the second training sample (two types of anchor samples), represents the label value of the original label provided by the m-th annotator on instance q when the true label value of instance p is i and the true label value of instance q is 1-j on instance q In this way, for each second training sample the model training system 110 can count the sum of the corresponding label values and, based on the ratio of this sum to the annotator, determine the above probability information of each second training sample Then, by calculating the average value of the probability information of all second training samples the transition matrix T corresponding to each pair of instances p, q can be determined (q,q) .

[0081] Continuing to refer to Figure 2 . After determining the label transfer information, at block 230, the model training system 110 determines the credibility of each of the multiple original labels based at least on the label transfer information corresponding to each pair of instances among the multiple instances.

[0082] As an example, the model training system 110 can evaluate the credibility of each original label based on the label transfer information through, such as, probability calculation, confidence evaluation, and other possible quantification metrics. The credibility reflects the confidence level of the model training system 110 in the accuracy of the original label.

[0083] ​​In some embodiments, for each of a plurality of training samples, the model training system 110 determines the credibility of each of the original labels corresponding to the training sample by calculating the posterior probability of each of the original labels corresponding to the training sample based on the label transfer information corresponding to each pair of instances in at least one pair of instances and the aggregated label of the training sample, where the aggregated label of the training sample is determined by performing an aggregation operation on the original labels corresponding to the training sample.

[0084] The posterior probability refers to the probability that a certain hypothesis (such as the credibility of the original label) holds after observing some data (such as the features and aggregated label of the training sample). Based on the label transfer information corresponding to each pair of instances in at least one pair of instances and the aggregated label of the training sample, the model training system 110 can calculate the posterior probability of each of the original labels corresponding to the training sample through statistical methods such as Bayes' theorem. It should be noted that the aggregated label can be determined in the manner described above, and the embodiments of the present disclosure will not be elaborated herein.

[0085] As an example, for each training sample (x, y) in the training dataset, the label transfer information can be used to determine the credibility of the original label y. The credibility can also be referred to as the confidence and its determination process can be expressed as A can be represented by the following formula (7):

[0086]

[0087] where represents the label transfer information T corresponding to a pair of instances 1, 1 (i.e., the first instance among all instances) (1,1) which is the probability that the original label labels instance 1 in the training sample as 1 when the true label value of instance 1 in the training sample is 1, represents the label transfer information T corresponding to a pair of instances 1, 2 (1,2) which is the probability that the original label labels instance 1 in the training sample as 1 when the true label value of instance 1 in the training sample is 1, represents the label transfer information T corresponding to a pair of instances 1, C (1,C) which is the probability that the original label labels instance C in the training sample as 1 when the true label value of instance 1 in the training sample is 1, and so on.

[0088] As an example, E can be represented by the following formula (8):

[0089]

[0090] where Indicates the aggregated label The label value corresponding to Instance 1 in Indicates the aggregated label The label value corresponding to Instance 2 in Indicates the aggregated label The label value corresponding to Instance C in, and so on. Indicates the label transition information T corresponding to Instance c,1 (c,1) Among them, when the true label value of Instance c in the training sample is 0, the probability that the original label labels Instance 1 in the training sample as 1 Indicates the label transition information T corresponding to Instance c,2 (c,2) Among them, when the true label value of Instance c in the training sample is 0, the probability that the original label labels Instance 2 in the training sample as 1 Indicates the label transition information T corresponding to Instance c, C (c,C) Among them, when the true label value of Instance c in the training sample is 0, the probability that the original label labels Instance C in the training sample as 1, and so on. By calculating the dot product of matrix A -1 and matrix E, the confidence of each original label can be quickly determined Thereby determining the credibility of each original label.

[0091] In block 240, the model training system 110 trains the first machine learning model (e.g., machine learning model 103) based on multiple training samples, multiple original labels, and the credibility of each original label. The first machine learning model is configured to detect whether there are multiple instances in the model input 121.

[0092] As an example, the first machine learning model can be any suitable architecture, such as a neural network, decision tree, support vector machine, etc. During training, the model training system 110 can consider the credibility of the original labels to adjust the model parameters to make the model more accurately predict the labels. The loss function is used to measure the difference between the prediction result of the first machine learning model and the true label. In the embodiments of the present disclosure, the model training system 110 can consider the credibility of the original labels to adjust the loss function, for example, giving a greater weight to the original labels with higher credibility to ensure that the first machine learning model pays more attention to these original labels during training. The model training system 110 continuously adjusts the model parameters of the first machine learning model through an iterative optimization algorithm to minimize the loss function until the convergence condition of model training is met.

[0093] As an example, the trained first machine learning model can be applied to various scenarios, such as image recognition, text classification, object detection, etc. In the image recognition task, the first machine learning model can detect multiple objects in the image and identify their categories; in the text classification task, the first machine learning model can identify multiple instances in the text and assign category labels to them.

[0094] In some embodiments, the model training system 110 trains the first machine learning model based on multiple training samples and multiple original labels, at least using the gradient descent algorithm, where in each iteration of the gradient descent algorithm, the loss function related to training is adjusted at least based on the credibility of each of the multiple original labels.

[0095] As an example, the model training system 110 can use the stochastic gradient descent algorithm (SGD) to iteratively optimize the loss function L'. As an example, the adjustment of the loss function L' can be represented by formulas (9) to (11).

[0096]

[0097] Where represents the loss value on sample x, n represents the number of training samples, g(x) is the first machine learning model, represents the loss value on instance c in the b-th iteration, g c (x) represents the predicted value of the first machine learning model for the label value on instance c, represents the label value on instance c in the aggregated label, l(g c (x), 1) represents that when , the predicted value g c (x) of the first machine learning model for instance c and The loss value brought by the difference between them, l(g c (x), 0) represents that when , the predicted value g c (x) of the first machine learning model for instance c and The loss value brought by the difference between them,

[0098] In the solution of the present disclosure, the label transfer information is used to identify the possibility of cross-label annotation errors between multiple original labels, and then the credibility of each original label is evaluated. During the model training process, by introducing the credibility of each original label, the machine learning model to be trained can make full use of multiple labels that may contain incorrect content, so as to learn the impact that annotation errors may bring.

[0099] It can be clearly understood from the various embodiments described above that, in view of the problem of label annotation in traditional machine learning model training, that is, it is difficult to obtain correct labels in actual applications, resulting in limited model training effects, the embodiments of the present disclosure identify cross-label annotation errors caused by the similarity of instances of different categories through label transfer information. Through label transfer information, the embodiments of the present disclosure can quickly calculate the confidence of each original label and affect the loss function in the model training process, so as to quickly obtain an unbiased loss, and then effectively alleviate the impact of cross-label annotation errors on model performance.

[0100] In the model inference stage after model training, the trained first machine learning model can be applied to perform recognition or classification of multiple instances (such as cats / dogs) on an input sample. The model input is a data sample corresponding to a modality, such as an image (or video, text, speech, etc.), and the model output can include predicted values corresponding to multiple instances, and each predicted value indicates whether the corresponding instance (such as a cat / dog) exists in the input sample.

[0101] Embodiments of the present disclosure also provide corresponding devices for implementing the above methods or processes. Figure 3 FIG. shows a schematic structural block diagram of a device 300 for model training according to some embodiments of the present disclosure. The device 300 can be implemented as or included in a model training system 110. Each module / component in the device 300 can be implemented by hardware, software, firmware, or any combination thereof.

[0102] Refer to Figure 3, the apparatus 300 includes a label acquisition module 310, a transfer information determination module 320, a credibility determination module 330, and a model training module 340. The label acquisition module 310 is configured to acquire a plurality of original labels corresponding to a plurality of training samples, where each training sample corresponds to at least two original labels, and each original label indicates whether there are multiple instances in the corresponding training sample. The transfer information determination module 320 is configured to determine, based on the plurality of original labels corresponding to the plurality of training samples, label transfer information corresponding to at least one pair of instances among the plurality of instances, where each pair of instances includes a first instance and a second instance, and the label transfer information corresponding to each pair of instances indicates: when the true label value of the first instance in the training sample is a first value and the true label value of the second instance is a third value, the possibility that the original label corresponding to the training sample mislabels the second instance in the training sample as a second value. The credibility determination module 330 is configured to determine the credibility of each of the plurality of original labels at least based on the label transfer information corresponding to each pair of instances among the plurality of instances. The model training module 340 is configured to train a first machine learning model based on the plurality of training samples, the plurality of original labels, and the credibility of each of the plurality of original labels, where the first machine learning model is configured to detect whether there are multiple instances in the model input.

[0103] In some embodiments, the transfer information determination module 320 is further configured to: determine a plurality of reference samples from the training sample set based on the original labels corresponding to the plurality of training samples, where each reference sample is determined as an anchor for the presence or absence of a single instance among the plurality of instances, or is determined as an anchor for the presence or absence of a pair of instances among the plurality of instances; and determine the label transfer information corresponding to at least one pair of instances among the plurality of instances based on the original labels corresponding to the plurality of reference samples.

[0104] In some embodiments, the transfer information determination module 320 is further configured to: train a second machine learning model based on the original labels corresponding to the plurality of training samples; use the trained second machine learning model to determine the predicted labels corresponding to the plurality of training samples, where the predicted labels indicate whether there are multiple instances in the corresponding training samples; and determine a plurality of reference samples from the training sample set based on the predicted labels corresponding to the plurality of training samples.

[0105] In some embodiments, the transfer information determination module 320 is further configured to: generate the aggregated labels corresponding to the plurality of training samples by performing an aggregation operation on the original labels corresponding to the plurality of training samples; and train a second machine learning model using the plurality of training samples and the aggregated labels corresponding to the plurality of training samples.

[0106] In some embodiments, the transfer information determination module 320 is further configured to: based on the predicted labels corresponding to the multiple training samples, determine multiple first training samples corresponding to each of the multiple instances from the multiple training samples, where the first training samples corresponding to a given instance among the multiple instances include: training samples for which the predicted value of the corresponding predicted label for the given instance is greater than a first threshold, and / or training samples for which the predicted value of the corresponding predicted label for the given instance is less than a second threshold, where the second threshold is less than the first threshold.

[0107] In some embodiments, the transfer information determination module 320 is further configured to: for the first instance and the second instance in each pair of instances among at least one pair of instances, in response to determining that the first instance and the second instance indicate the same instance, determine the label transfer information corresponding to the pair of instances based on the likelihood that the original label corresponding to the first training sample mislabels the second instance as a second value when the true label value of the first instance is a first value.

[0108] In some embodiments, the transfer information determination module 320 is further configured to: based on the predicted labels corresponding to the multiple training samples, determine multiple second training samples corresponding to each pair of instances among at least one pair of instances from the multiple training samples, where the multiple second training samples corresponding to each pair of instances among at least one pair of instances include at least one of the following: for the first instance and the second instance in each pair of instances, training samples for which the predicted value of the corresponding predicted label for the first instance is greater than the first threshold and the predicted value for the second instance is also greater than the first threshold, training samples for which the predicted value of the corresponding predicted label for the first instance is greater than the first threshold and the predicted value for the second instance is less than the second threshold, training samples for which the predicted value of the corresponding predicted label for the first instance is less than the second threshold and the predicted value for the second instance is greater than the first threshold, and / or training samples for which the predicted value of the corresponding predicted label for the first instance is less than the second threshold and the predicted value for the second instance is also less than the second threshold.

[0109] In some embodiments, the transfer information determination module 320 is further configured to: for the first instance and the second instance in each pair of instances among at least one pair of instances, in response to determining that the first instance and the second instance indicate different instances, determine the label transfer information corresponding to the pair of instances based on the likelihood that the original label corresponding to the second training sample mislabels the second instance as a second value when the true label value of the first instance is a first value and the true label value of the second instance is a third value.

[0110] In some embodiments, the label transfer information is represented by a transfer matrix.

[0111] In some embodiments, the trust level determination module 330 is further configured to: for each of the multiple training samples, based on the label transfer information corresponding to each pair of instances in at least one pair of instances and the aggregated label of the training sample, determine the trust level of each of the original labels corresponding to the training sample by calculating the posterior probability of each of the original labels corresponding to the training sample, where the aggregated label of the training sample is determined by performing an aggregation operation on the original labels corresponding to the training sample.

[0112] In some embodiments, the model training module 340 is further configured to: based on the multiple training samples and the multiple original labels, train a first machine learning model at least using the gradient descent algorithm, where in each iteration of the gradient descent algorithm, adjust the loss function related to training in that iteration at least based on the trust level of each of the multiple original labels.

[0113] In some embodiments, different original labels corresponding to the same training sample are labeled by different annotators.

[0114] Figure 4 A block diagram of an electronic device 400 in which one or more embodiments of the present disclosure may be implemented is shown. The electronic device 400 may be used, for example, to implement the model training system 110 as shown in Figure 1 or the device 300 as shown in Figure 3 . It should be understood that Figure 4 the electronic device 400 shown is merely exemplary and should not constitute any limitation to the functions and scope of the embodiments described herein.

[0115] Referring to Figure 4 , the electronic device 400 is in the form of a general-purpose electronic device. The components of the electronic device 400 may include, but are not limited to, one or more processors or a processor 410, a memory 420, a storage device 430, one or more communication units 440, one or more input devices 450, and one or more output devices 460. The processor 410 may be an actual or virtual processor and is capable of performing various processes according to the programs stored in the memory 420. In a multi-processor system, multiple processors execute computer-executable instructions in parallel to improve the parallel processing ability of the electronic device 400.

[0116] The electronic device 400 generally includes multiple computer storage media. Such media can be any available media accessible to the electronic device 400, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 420 can be volatile memory (such as registers, caches, random access memory (RAM)), non-volatile memory (such as read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 430 can be removable or non-removable media and can include machine-readable media, such as a flash drive, a magnetic disk, or any other media that can be capable of storing information and / or data and can be accessed within the electronic device 400.

[0117] The electronic device 400 can further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in Figure 4 it, a disk drive for reading from or writing to a removable, non-volatile magnetic disk (such as a "floppy disk") and an optical disk drive for reading from or writing to a removable, non-volatile optical disk can be provided. In these cases, each drive can be connected to a bus (not shown) by one or more data media interfaces. The memory 420 can include a computer program product 425 having one or more program modules that are configured to execute the various methods or actions of the various embodiments of the present disclosure.

[0118] The communication unit 440 enables communication with other electronic devices via a communication medium. Additionally, the functions of the components of the electronic device 400 can be implemented by a single computing cluster or multiple computer machines that can communicate via a communication connection. Thus, the electronic device 400 can operate in a networked environment using a logical connection with one or more other servers, network personal computers (PCs), or another network node.

[0119] The input device 450 can be one or more input devices, such as a mouse, a keyboard, a trackball, etc. The output device 460 can be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 400 can also communicate with one or more external devices (not shown) as needed via the communication unit 440, the external devices such as storage devices, display devices, etc., communicate with one or more devices that enable a user to interact with the electronic device 400, or communicate with any device that enables the electronic device 400 to communicate with one or more other electronic devices (such as a network card, a modem, etc.). Such communication can be performed via an input / output (I / O) interface (not shown).

[0120] According to an exemplary implementation of the present disclosure, there is provided a computer-readable storage medium having computer-executable instructions stored thereon, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, there is also provided a computer program product, the computer program product being tangibly stored on a non-transitory computer-readable medium and including computer-executable instructions, and the computer-executable instructions being executed by a processor to implement the method described above.

[0121] Aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of methods, apparatuses, devices, and computer program products according to the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0122] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, when executed by the processor of the computer or other programmable data processing apparatus, create a means for implementing the functions / acts specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which instructions cause a computer, a programmable data processing apparatus, and / or other devices to operate in a particular manner, so that the computer-readable medium storing the instructions includes a manufacture comprising instructions for implementing various aspects of the functions / acts specified in one or more blocks of the flowchart and / or block diagram.

[0123] The computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in one or more blocks of the flowchart and / or block diagram.

[0124] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various implementations of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the blocks may occur in a different order than noted in the accompanying drawings. For example, two consecutive blocks may in fact be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or actions, or by a combination of dedicated hardware and computer instructions.

[0125] The implementations of the present disclosure have been described above. The above description is exemplary, not exhaustive, and is not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The determination of the terms used herein is intended to best explain the principles of the implementations, the practical application, or the improvement of the technology in the market, or to enable other ordinary skilled in the art to understand the various implementation manners disclosed herein.

Claims

1. A model training method, comprising: Acquire multiple original labels corresponding to multiple training samples, wherein each training sample corresponds to at least two original labels, and each original label indicates whether there are multiple instances in the corresponding training sample; Determine, based on a plurality of original labels corresponding to the plurality of training samples, label transfer information corresponding to at least one pair of instances among the plurality of instances, wherein each pair of instances includes a first instance and a second instance, and the label transfer information corresponding to each pair of instances indicates: when a true value label value of the first instance in the training sample is a first value and a true value label value of the second instance is a third value, a possibility that the original label corresponding to the training sample incorrectly labels the second instance as a second value; Determine the credibility of each of the plurality of original labels based at least on the label transfer information corresponding to each pair of instances in the plurality of instances; as well as Based on the multiple training samples, the multiple original labels, and the credibility of each of the multiple original labels, a first machine learning model is trained, wherein the first machine learning model is configured to detect whether the multiple instances exist in the model input.

2. The method according to claim 1, wherein determining, based on the multiple original labels, label transfer information corresponding to at least one pair of instances among the multiple instances comprises: Based on the original labels corresponding to the plurality of training samples, a plurality of reference samples are determined from the training sample set, wherein each reference sample is determined as an anchor point for the presence or absence of a single instance in the plurality of instances, or is determined as an anchor point for the presence or absence of a pair of instances in the plurality of instances; as well as Based on the original labels corresponding to the multiple reference samples, label transfer information corresponding to at least one pair of instances among the multiple instances is determined.

3. The method according to claim 2, wherein determining a plurality of reference samples from the training sample set comprises: Training a second machine learning model based on the original labels corresponding to each of the multiple training samples; Determine, using the trained second machine learning model, prediction labels corresponding to each of the plurality of training samples, wherein the prediction labels indicate whether the plurality of instances exist in the corresponding training samples; as well as Based on the predicted labels corresponding to each of the multiple training samples, multiple reference samples are determined from the training sample set.

4. The method according to claim 3, wherein based on the original labels corresponding to each of the plurality of training samples, training the second machine learning model comprises: Generate aggregated labels corresponding to the multiple training samples respectively by performing an aggregation operation on the original labels corresponding to the multiple training samples respectively; as well as The second machine learning model is trained by using the multiple training samples and the aggregated labels corresponding to each of the multiple training samples.

5. The method according to claim 3, wherein determining a plurality of reference samples from the training sample set based on the predicted labels corresponding to each of the plurality of training samples comprises: Based on the predicted labels corresponding to the plurality of training samples, a plurality of first training samples corresponding to each of the plurality of instances are determined from the plurality of training samples, The first training samples corresponding to a given instance among the multiple instances include: training samples whose corresponding prediction labels have a prediction value for the given instance greater than a first threshold, and / or training samples whose corresponding prediction labels have a prediction value for the given instance less than a second threshold, wherein the second threshold is less than the first threshold.

6. The method according to claim 5, wherein determining the label transfer information corresponding to at least one pair of instances among the multiple instances based on the original labels corresponding to the multiple reference samples comprises: For a first instance and a second instance in each pair of instances in the at least one pair of instances, In response to determining that the first instance and the second instance indicate the same instance, based on the possibility that the original label corresponding to the first training sample incorrectly labels the second instance as the second value when the true value label value of the first instance is the first value, the label transfer information corresponding to the pair of instances is determined.

7. The method according to claim 3, wherein determining a plurality of reference samples from the training sample set based on the predicted labels corresponding to each of the plurality of training samples further comprises: Based on the predicted labels corresponding to the plurality of training samples, respectively, determining a plurality of second training samples corresponding to each pair of instances in the at least one pair of instances from the plurality of training samples, The plurality of second training samples corresponding to each pair of instances in the at least one pair of instances include at least one of the following: for the first instance and the second instance in each pair of instances, training samples whose prediction value for the first instance is greater than a first threshold and whose prediction value for the second instance is also greater than the first threshold, a training sample in which the prediction value of the corresponding prediction label for the first instance is greater than the first threshold and the prediction value for the second instance is less than a second threshold, wherein the second threshold is less than the first threshold, training samples whose predicted value for the first instance is less than the second threshold and whose predicted value for the second instance is greater than the first threshold, and / or The prediction value of the corresponding prediction label for the first instance is smaller than the second threshold, and the prediction value for the second instance is also smaller than the second threshold for the training sample.

8. The method according to claim 7, wherein determining the label transfer information corresponding to at least one pair of instances among the multiple instances based on the original labels corresponding to the multiple reference samples comprises: For a first instance and a second instance in each pair of instances in the at least one pair of instances, In response to determining that the first instance and the second instance indicate different instances, the label transfer information corresponding to the pair of instances is determined based on the possibility that the original label corresponding to the second training sample incorrectly labels the second instance as the second value when the true label value of the first instance is the first value and the true label value of the second instance is the third value.

9. The method according to claim 1, wherein the label transfer information is represented by a transfer matrix.

10. The method according to claim 1, wherein determining the credibility of each of the plurality of original tags comprises: For each training sample in the plurality of training samples, Based on the label transfer information corresponding to each pair of instances in the at least one pair of instances and the aggregated label of the training sample, the credibility of each original label corresponding to the training sample is determined by calculating the posterior probability of each original label corresponding to the training sample, The aggregated label of the training sample is determined by performing an aggregation operation on the original label corresponding to the training sample.

11. The method of claim 1 , wherein training the first machine learning model comprises: Based on the multiple training samples and the multiple original labels, the first machine learning model is trained using at least a gradient descent algorithm, wherein in each iteration of the gradient descent algorithm, the loss function related to the training in that iteration is adjusted based at least on the credibility of each of the multiple original labels.

12. The method according to claim 1, wherein different original labels corresponding to the same training sample are annotated by different annotators.

13. A device for model training, comprising: A label acquisition module is configured to acquire a plurality of original labels corresponding to a plurality of training samples, wherein each training sample corresponds to at least two original labels, and each original label indicates whether there are multiple instances in the corresponding training sample; a transfer information determination module, configured to determine, based on a plurality of original labels corresponding to the plurality of training samples, label transfer information corresponding to at least one pair of instances among the plurality of instances, wherein each pair of instances includes a first instance and a second instance, and the label transfer information corresponding to each pair of instances indicates: when the true value label value of the first instance in the training sample is a first value and the true value label value of the second instance is a third value, the possibility that the original label corresponding to the training sample incorrectly labels the second instance in the training sample as a second value; a credibility determination module, configured to determine the credibility of each of the plurality of original labels based at least on the label transfer information corresponding to each pair of instances in the plurality of instances; as well as The model training module is configured to train a first machine learning model based on the multiple training samples, the multiple original labels, and the credibility of each of the multiple original labels, wherein the first machine learning model is configured to detect whether the multiple instances exist in the model input.

14. An electronic device comprising: at least one processor; as well as At least one memory, the at least one memory is coupled to the at least one processor and stores instructions for execution by the at least one processor, the instructions causing the electronic device to perform the method according to any one of claims 1 to 12 when executed by the at least one processor.

15. A computer-readable storage medium having computer-executable instructions stored thereon, wherein the computer-executable instructions can be executed by a processor to implement the method according to any one of claims 1 to 12.

16. A computer program product comprising computer executable instructions, wherein the computer executable instructions, when executed by a processor, implement the method according to any one of claims 1 to 12.

Images