Multi-modal recognition method, device and equipment based on noise label, storage medium and program product

Abstract

The application relates to a multi-modal recognition method, device and equipment based on noise labels, a storage medium and a program product. The method comprises the following steps: acquiring noise sample data sets of a visual language model, wherein the noise sample data sets comprise a plurality of image samples and sample labels corresponding to the plurality of image samples respectively; dividing the sample labels into trusted labels and untrusted labels according to the semantic trust boundary of the sample labels; performing prompt learning on the visual language model according to the image samples of the trusted labels and the image samples of the untrusted labels, so as to optimize the visual language model; and using the optimized visual language model to recognize an image to be recognized in a downstream task and determining the category of the image to be recognized. The method can improve the recognition accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a multi-modal recognition method and device based on noise labels, equipment, a storage medium and a program product. BACKGROUND

[0002] Currently, prompt learning of a visual language model (VLM) establishes an efficient new paradigm for transfer learning by separating parameter adaptation from feature extraction. In particular, the potential in low-resource scenarios has been demonstrated in a variety of tasks, including pathological slide classification, hyperspectral object recognition, and other scenarios.

[0003] In related technologies, the instant tuning achieved by prompt learning exhibits excellent performance in low-resource tasks, and the fundamental reason is the correctness of the supervision signal. However, in actual applications, downstream datasets are often affected by noisy annotations caused by automated data collection or human errors, which poses a major challenge to instant tuning with sparse parameters, resulting in a low recognition accuracy of the visual language model trained by prompt learning. SUMMARY

[0004] Therefore, it is necessary to provide a multi-modal recognition method, device, equipment, storage medium and program product based on noise labels to improve the recognition accuracy.

[0005] In a first aspect, the present application provides a multi-modal recognition method based on noise labels, comprising:

[0006] obtaining a noise sample dataset of a visual language model, the noise sample dataset comprising a plurality of image samples and sample labels corresponding to the plurality of image samples, respectively;

[0007] According to the semantic trust boundary of the sample label, the sample label is divided into trusted labels and untrusted labels;

[0008] According to the image samples of the trusted labels and the image samples of the untrusted labels, prompt learning is performed on the visual language model to optimize the visual language model;

[0009] using the optimized visual language model to recognize the to-be-recognized image in the downstream task, and determining the category of the to-be-recognized image;

[0010] The visual language model is a model pre-trained to align image and text representations in a shared embedding space; the prompt learning is used to adjust a context vector in the visual language model through a gradient of a trusted label and a gradient of a non-trusted label; the gradient of the non-trusted label is generated after projection optimization according to the gradient of the trusted label; and the context vector is used to generate a text prompt of each category in the downstream task to map the text prompt to the text representation.

[0011] In one of the embodiments, before dividing the sample labels into trusted labels and non-trusted labels according to a semantic trust boundary of the sample labels, the method further comprises:

[0012] performing zero-shot prediction on a sample image of the sample label using the visual language model to obtain a zero-shot prediction result;

[0013] determining the semantic trust boundary of the sample label according to the zero-shot prediction result.

[0014] In one of the embodiments, the determining the semantic trust boundary of the sample label according to the zero-shot prediction result comprises:

[0015] determining a prediction category set corresponding to the sample label according to the zero-shot prediction result, the prediction category set including K categories with the highest probabilities obtained by zero-shot prediction of the image sample corresponding to the sample label using the visual language model, K being an integer greater than or equal to zero;

[0016] determining the semantic trust boundary of the sample label according to the prediction category set corresponding to the sample label; wherein a sample label within the semantic trust boundary is within the prediction category set, and a sample label outside the semantic trust boundary is not within the prediction category set.

[0017] In one of the embodiments, before the determining the prediction category set corresponding to the sample label according to the zero-shot prediction result, the method further comprises:

[0018] determining the precision and recall of the prediction category set corresponding to different K values;

[0019] determining the probability index score of the prediction category set corresponding to different K values according to the precision and recall of the prediction category set corresponding to different K values;

[0020] determining the K value corresponding to the highest probability index score as the number of categories included in the prediction category set.

[0021] In one of the embodiments, the optimizing the visual language model comprises:

[0022] determining an average gradient of the trust labels;

[0023] determining a cosine similarity corresponding to the gradient of the untrust label according to the average gradient of the trust labels;

[0024] optimizing the gradient of the untrust label according to the cosine similarity corresponding to the gradient of the untrust label;

[0025] optimizing the visual language model using the optimized gradient of the untrust label and the gradient of the trust label.

[0026] In one embodiment, the optimizing the gradient of the untrust label according to the cosine similarity corresponding to the gradient of the untrust label comprises:

[0027] if the cosine similarity is less than zero, projecting the gradient of the untrust label into a trust alignment space corresponding to the average gradient of the trust labels;

[0028] optimizing the gradient of the untrust label by removing the component of the gradient of the untrust label in the direction of the average gradient in the trust alignment space.

[0029] In a second aspect, the present application further provides a multi-modal recognition device based on noise labels, comprising:

[0030] an acquisition module configured to acquire a noise sample dataset of a visual language model, wherein the noise sample dataset comprises a plurality of image samples and sample labels corresponding to the plurality of image samples respectively;

[0031] a division module configured to divide the sample labels into trust labels and untrust labels according to a semantic trust boundary of the sample labels;

[0032] a learning module configured to respectively perform prompt learning on the visual language model according to image samples of the trust labels and image samples of the untrust labels, so as to optimize the visual language model;

[0033] a recognition module configured to recognize a to-be-recognized image in a downstream task using the optimized visual language model, and determine a category of the to-be-recognized image.

[0034] The visual language model is a model pre-trained to align image and text representations in a shared embedding space; the prompt learning is used to adjust a context vector in the visual language model through a gradient of a trusted label and a gradient of a non-trusted label; the gradient of the non-trusted label is generated after projection optimization according to the gradient of the trusted label; and the context vector is used to generate a text prompt of each category in the downstream task to map the text prompt to the text representation.

[0035] In one of the embodiments, the dividing module is further configured to perform zero-shot prediction on the sample image of the sample label by using the visual language model to obtain a zero-shot prediction result; and determine the semantic trust boundary of the sample label according to the zero-shot prediction result.

[0036] In one of the embodiments, the dividing module is further configured to determine a predicted category set corresponding to the sample label according to the zero-shot prediction result, the predicted category set including K categories with the highest probability obtained by zero-shot prediction of the image sample corresponding to the sample label by using the visual language model, K being an integer greater than or equal to zero; determine the semantic trust boundary of the sample label according to the predicted category set corresponding to the sample label; and the sample label within the predicted category set is within the semantic trust boundary, and the sample label not within the predicted category set is outside the semantic trust boundary.

[0037] In one of the embodiments, the dividing module is further configured to determine the precision and recall rate of the predicted category set corresponding to different K values; determine the probability index score of the predicted category set corresponding to different K values according to the precision and recall rate of the predicted category set corresponding to different K values; and determine the number of categories included in the predicted category set according to the K value corresponding to the highest probability index score.

[0038] In one of the embodiments, the learning module is further configured to determine the average gradient of the trusted label; determine the cosine similarity corresponding to the gradient of the non-trusted label according to the average gradient of the trusted label; optimize the gradient of the non-trusted label according to the cosine similarity corresponding to the gradient of the non-trusted label; and optimize the visual language model by using the optimized gradient of the non-trusted label and the gradient of the trusted label.

[0039] In one of the embodiments, the learning module is further configured to project the gradient of the non-trusted label into a trust alignment space corresponding to the average gradient of the trusted label if the cosine similarity is less than zero; and optimize the gradient of the non-trusted label by removing the component of the gradient of the non-trusted label in the direction of the average gradient in the trust alignment space.

[0040] In a third aspect, the present application also provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the noise label based multi-modal recognition method of the first aspect.

[0041] In a fourth aspect, the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the noise label based multi-modal recognition method of the first aspect.

[0042] In a fifth aspect, the present application also provides a computer program product comprising a computer program, and the computer program is executed by a processor to implement the noise label based multi-modal recognition method of the first aspect.

[0043] The noise label based multi-modal recognition method, device, equipment, storage medium and program product, obtain a noise sample data set of a visual language model, the noise sample data set comprises a plurality of image samples and sample labels corresponding to the plurality of image samples respectively; according to a semantic trust boundary of the sample labels, the sample labels are divided into trusted labels and untrusted labels; according to the image samples of the trusted labels and the image samples of the untrusted labels, prompt learning is performed on the visual language model respectively, so as to optimize the visual language model; using the optimized visual language model, an image to be recognized in a downstream task is recognized to determine a category of the image to be recognized; wherein the visual language model is a model for aligning image and text representations in a shared embedding space through pre-training; the prompt learning is used to adjust a context vector in the visual language model through a gradient of the trusted labels and a gradient of the untrusted labels; the gradient of the untrusted labels is generated after projection optimization according to the gradient of the trusted labels; the context vector is used to generate a text prompt of each category in the downstream task, so as to map the text prompt to a text representation. Since the trusted labels and the untrusted labels are distinguished through the semantic trust boundary of the sample labels, and the gradient of the untrusted labels is projected and optimized according to the gradient of the trusted labels in the prompt learning, the optimization drift caused by the untrusted labels is suppressed, and the accuracy of recognizing the category of the image using the visual language model is improved. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the description of the embodiments of the present application or the related art will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other related drawings can also be obtained without creative labor.

[0045] Figure 1An application environment diagram of a multi-modal recognition method based on noise labels provided by an embodiment of the present application is provided;

[0046] Figure 2 A flowchart of a multi-modal recognition method based on noise labels provided by an embodiment of the present application is provided;

[0047] Figure 3 A flowchart of another multi-modal recognition method based on noise labels provided by an embodiment of the present application is provided;

[0048] Figure 4 A flowchart of still another multi-modal recognition method based on noise labels provided by an embodiment of the present application is provided;

[0049] Figure 5 A structural block diagram of a multi-modal recognition device based on noise labels provided by an embodiment of the present application is provided;

[0050] Figure 6 An internal structure diagram of a computer device provided by an embodiment of the present application is provided. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical scheme and advantages of the present application clearer, further detailed description of the present application will be given below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0052] Firstly, the related art will be described below.

[0053] Prompt learning of visual language model establishes an efficient new paradigm of transfer learning by separating parameter adaptation from feature extraction. For example, due to the inherent characteristics of the dual-encoder architecture in the visual language model, i.e., freezing the image encoder preserves strong generalization ability, while prompt learning re-maps downstream classes to the pre-trained semantic space in semantics, so that the Context Optimization (CoOp) prompt tuning can only optimize the text prompt of the visual language model. Based on this, the potential of prompt learning of visual language model in low-resource scenarios has been proven in a variety of tasks, including pathological slide classification, hyperspectral object recognition, etc.

[0054] However, the instant tuning achieved by prompt learning exhibits excellent performance in low-resource tasks, and the fundamental reason lies in the correctness of the supervision signal. In practical applications, downstream datasets are often affected by noisy annotations caused by automated data collection or human errors, which poses a major challenge to instant tuning with sparse parameters. When introducing asymmetric label noise, the performance of CoOp will rapidly decrease with the increase of noise level.

[0055] It should be understood that in the contrastive language-image pre-training (CLIP) of the visual language model, the image and text encoders of the visual language model are in a frozen state even if they encode strong prior knowledge, and therefore the text prompt, as the only trainable component, is insufficient to distinguish semantic bias or correct misplacement in the pre-training space when there is label noise, resulting in confusion of inter-class relationships and misdirection of the optimization process of the visual language model. In addition, the traditional cross-entropy loss function treats noisy samples and clean samples equally, further amplifying the negative impact of mislabeled data. Therefore, improving the robustness of rapid tuning under noisy supervision has become a key bottleneck for its practical application.

[0056] In related technologies, there are two anti-noise training methods for noisy labels. In the first way, samples with unstable predictions are directly filtered out through confidence modeling or uncertainty estimation, and in the second way, a robust loss function such as generalized cross-entropy (GCE) is used to mitigate noise interference by assigning lower gradient weights to low-confidence samples. However, the anti-noise training methods in related technologies ignore the semantic structure of the visual language model, especially the inter-class space constructed by CLIP.

[0057] The present application provides a multi-modal recognition method, device, equipment, storage medium and program product based on noisy labels, which distinguishes trusted labels and untrusted labels through the semantic trust boundary of noisy labels, and then projects and optimizes the gradient of untrusted labels according to the gradient of trusted labels in prompt learning, thereby suppressing the optimization drift caused by untrusted labels, and further improving the accuracy of identifying the class of an image using a visual language model.

[0058] The application scenarios of the multi-modal recognition method based on noisy labels related to the embodiments of the present application will be described below.

[0059] The multi-modal recognition method based on noisy labels provided by the embodiments of the present application can be applied in an application environment as shown in Figure 1 The terminal 102 communicates with the server 104 through a network. The data storage system can store the data required to be processed by the server 104. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers.

[0060] The server 104 can first obtain a noise sample data set of the visual language model, the noise sample data set including a plurality of image samples and a plurality of sample labels corresponding to the plurality of image samples respectively. Second, the server 104 divides the sample labels into trusted labels and untrusted labels according to the semantic trust boundary of the sample labels. Third, the server 104 performs prompt learning on the visual language model according to the image samples of the trusted labels and the image samples of the untrusted labels, to optimize the visual language model. Finally, the terminal 102 sends an image to be recognized in a downstream task to the server, and the server 104 uses the optimized visual language model to recognize the image to be recognized in the downstream task and determine the category of the image to be recognized.

[0061] The visual language model is a model that aligns image and text representations in a shared embedding space through pre-training; the prompt learning is used to adjust a context vector in the visual language model through a gradient of the trusted labels and a gradient of the untrusted labels; the gradient of the untrusted labels is generated after projection optimization according to the gradient of the trusted labels; and the context vector is used to generate a text prompt of each category in the downstream task to map the text prompt to the text representation.

[0062] The terminal 102 can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things device can be a smart speaker, a smart television, a smart air conditioner, a smart vehicle-mounted device, a projection device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 104 can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0063] In an exemplary embodiment, as shown in Figure 2 , a multi-modal recognition method based on noise labels is provided. The multi-modal recognition method based on noise labels is applied to the server in Figure 1 for example, and includes S201-S204.

[0064] S201, obtaining a noise sample data set of a visual language model.

[0065] It should be understood that the embodiments of the present application are not limited to the above noise sample dataset, and in some embodiments, the above noise sample dataset is a noise sample dataset corresponding to a downstream task, and the downstream task is a specific application task that needs to be solved on the basis of a pre-trained model in machine learning. For example, the downstream task includes a case slide classification task, a hyperspectral object recognition task, and the like.

[0066] In some embodiments, the noise sample dataset includes a plurality of image samples and a plurality of sample labels corresponding to the plurality of image samples, respectively. The sample label is a label that may be incorrect, inaccurate, or inconsistent with the true attribute of the sample, and the sample label is used to identify the category to which the corresponding sample image belongs.

[0067] For example, if the sample dataset with real labels contains N samples, the sample dataset with real labels As shown in formula (1), when the real sample is changed to a noise sample (i.e., an image sample with a non-trusted label), the noise sample dataset As shown in formula (2).

[0068] (1)

[0069] (2)

[0070] wherein, is the i-th sample image, N is the total number of samples, is the real label corresponding to the i-th sample image, L is the total number of categories in the downstream task, is the noise label (i.e., the non-trusted label) corresponding to the i-th sample image.

[0071] It should be understood that a variety of reasons can cause the label to be damaged, thereby causing the real label to become a noise label. For example, the real label becoming a noise label can be represented by formula (3).

[0072] (3)

[0073] wherein, is the probability that the sample with a real label i is incorrectly labeled as category j, is the real label, is the noise label.

[0074] In some embodiments, the noise labels can include symmetric noise labels and asymmetric noise labels. Among them, the symmetric noise labels can be labels with the same probability of being assigned as noise for all incorrect categories. The asymmetric noise labels can be labels simulating real confusion between semantically or visually similar categories, where label flipping only occurs between specific pairs of categories.

[0075] Exemplarily, the transition matrix of the symmetric noise label can be shown as formula (4), and the transition matrix of the asymmetric noise label can be shown as formula (5).

[0076] (4)

[0077] (5)

[0078] wherein, is the noise rate. When , the label is clean; when , the label is completely randomized. represents a class visually similar to class i.

[0079] It should be understood that the noise rate is used to characterize the degree or proportion of noise present in the noise sample dataset. The noise labels in the above noise sample dataset can be symmetric noise labels and asymmetric noise labels, and the noise level can be set according to the actual situation, for example, it can be 0.125, 0.25, 0.75, etc.

[0080] S202, according to the semantic trust boundary of the sample label, the sample label is divided into trusted labels and untrusted labels.

[0081] In this step, after the server obtains the noise sample dataset of the visual language model, the sample label can be divided into trusted labels and untrusted labels according to the semantic trust boundary of the sample label.

[0082] Among them, the semantic trust boundary of the sample label is a boundary constructed by the closeness between the sample label and the zero sample prediction. The sample label within the semantic trust boundary can be a trusted label, and the sample label outside the semantic trust boundary can be an untrusted label.

[0083] Among them, the trusted label can be trusted as a real label; the untrusted label cannot be trusted as a real label, that is, the untrusted label can be considered as the above noise label.

[0084] In some embodiments, before dividing the sample labels into trusted labels and untrusted labels according to the semantic trust boundary of the sample labels, the server can first perform zero-shot prediction on the sample images of the sample labels using the visual language model to obtain a zero-shot prediction result. Subsequently, the server determines the semantic trust boundary of the sample labels according to the zero-shot prediction result.

[0085] wherein the zero-shot prediction can be a prediction or classification of the class of the downstream task by the visual language model only through prior information without training of the downstream task.

[0086] It should be understood that in the zero-shot prediction, even if the top-1 prediction is incorrect, the true label often appears in the top-ranked predictions, and therefore the prediction class set of the K class groups with the highest probabilities obtained through the zero-shot prediction can be used to construct the semantic trust boundary.

[0087] In some embodiments, the server can first determine the prediction class set corresponding to the sample labels according to the zero-shot prediction result. Subsequently, the server determines the semantic trust boundary of the sample labels according to the prediction class set corresponding to the sample labels.

[0088] wherein the prediction class set includes the K classes with the highest probabilities obtained through the zero-shot prediction of the image sample corresponding to the sample label by the visual language model, and K is an integer greater than or equal to zero. For example, the prediction class set can be a Top-K class set including the K classes with the highest probabilities.

[0089] Correspondingly, the sample labels within the semantic trust boundary in the prediction class set are trusted, and the sample labels outside the semantic trust boundary are not trusted.

[0090] For example, if the image sample is the probability distribution obtained through the zero-shot prediction is and the Top-K class set is which can be shown in formula (6).

[0091] (6)

[0092] For example, before training, the static template "[CLASS] photo" can be used as a text prompt for Top-K evaluation to establish an initial semantic consistency reference. For example, after the training starts, we switch to a learnable context vector generated text prompt, which is constantly updated during the training of the visual language model, and can be shown in formula (7).

[0093] (7)

[0094] wherein m e [1, M], may be the mth learnable context vector, which is constantly updated during the training process. is a vector of class names for the downstream task.

[0095] For example, as the visual language model improves its representation ability during the training process, the accuracy of semantic consistency estimation will also improve. If the sample label of the image sample is contained in the Top-K class set , it is considered that the image sample is semantically consistent with the model prediction, and it is divided into a trusted label. If the sample label of the image sample is not contained in the Top-K class set , it is considered that the image sample is semantically inconsistent with the model prediction, and it can be used as a non-trusted label.

[0096] wherein K is a hyperparameter that controls the number of candidate classes. The Top-K class set corresponds to the hyperparameter K, which effectively acts as a classification threshold in semantic verification. Therefore, the selection of K directly affects the distinction between trusted labels and non-trusted labels. The hyperparameter K can be determined by the highest value of the probability indicator score F1.

[0097] In some embodiments, the server can first determine the precision and recall corresponding to the prediction class set of different K values. Then, the server determines the probability indicator score corresponding to the prediction class set of different K values according to the precision and recall corresponding to the prediction class set of different K values. Finally, the server determines the K value corresponding to the highest probability indicator score as the number of classes contained in the prediction class set.

[0098] The precision of the prediction class set of different K values may be determined by formula (8), the recall of the prediction class set of different K values may be determined by formula (9). After determining the precision and recall, the probability indicator score corresponding to the prediction class set of different K values can be determined by formula (10), and the K value corresponding to the highest probability indicator score can be selected as the optimal K by formula (11) to achieve a balanced trade-off, i.e. the number of classes contained in the prediction class set.

[0099] (8)

[0100] (9)

[0101] (10)

[0102] (11)

[0103] where K is a hyper-parameter that controls the number of candidate classes, representing the number of classes contained in the predicted class set; precision, recall, probability score, untrusted label, sample label, noisy sample dataset.

[0104] In some embodiments, after determining K, a soft label can also be assigned to the sample label. Exemplarily, the soft label can be assigned to the sample label according to formula (12).

[0105] (12)

[0106] where, soft label, L is the total number of classes of the downstream task, smoothing factor, K is a hyper-parameter that controls the number of candidate classes, representing the number of classes contained in the predicted class set.

[0107] Exemplarily, the soft label is initialized by assigning uniform probabilities to the Top-K classes and zero probabilities to all other classes. At this time, a smoothing factor can be further applied to obtain the final label distribution. The probability mass of the Top-K classes is and the probability mass of the remaining classes is . In this way, the robustness can be enhanced by preventing excessive trust in potentially noisy targets.

[0108] The false positive rate of the semantic trust boundary under label division of the sample label is described below.

[0109] It should be understood that the false positive rate described above can be the probability of misclassifying the untrusted label as the trusted label under the semantic trust boundary.

[0110] Exemplarily, an untrusted label setting is controlled by a conversion matrix , where the entry , and the total untrusted rate . Correspondingly, the limit of the false positive rate can be shown in formula (13).

[0111] (13)

[0112] S203, respectively, prompt learning is performed on the visual language model according to the image samples of the trust label and the image samples of the non-trust label, so as to optimize the visual language model.

[0113] In this step, after the server divides the sample label into a trust label and a non-trust label according to the semantic trust boundary of the sample label, respectively, prompt learning is performed on the visual language model according to the image samples of the trust label and the image samples of the non-trust label, so as to optimize the visual language model.

[0114] Wherein, the visual language model is a model that aligns image and text representations in a shared embedding space through pre-training; the prompt learning is used to adjust the context vector in the visual language model through the gradient of the trust label and the gradient of the non-trust label; the context vector is used to generate a text prompt describing each class in the downstream task, so as to map the text prompt to the text representation.

[0115] For example, the visual language model aligns image and text representations in a shared embedding space through large-scale contrastive learning on pairs of image-text data during pre-training. In the downstream classification task, prompt learning optimizes the visual language model by introducing a set of learnable context vectors to construct text prompts describing each class in the downstream task. The context vector can be as shown in formula (7). The sample image x is processed by the image encoder in the visual language model to obtain the visual feature ; the text prompt is processed by the text encoder in the visual language model to obtain the corresponding text feature . The cosine similarity between the two features is used to measure the semantic alignment between the sample image and the text representation. Then, the probability of classifying the sample image x into class i by the visual language model is as shown in formula (14).

[0116] (14)

[0117] Wherein, is the probability of classifying the sample image x into class i, is the visual feature of the sample image, is the text feature of the text prompt , L is the total number of classes of the downstream task, and T is a temperature parameter. T is used to control the smoothness of the output. In the L-way classification setting, the visual language model generates the final prediction vector as .

[0118] It should be understood that, in the optimization process, the prompt learning only optimizes the context vector, while the image encoder and the text encoder are kept in a frozen state.

[0119] In some embodiments, the gradient of the untrusted label is generated after projection optimization according to the gradient of the trusted label.

[0120] In the embodiments of the present application, the trusted label and the untrusted label can be effectively identified from the sample label through the semantic trust boundary. The trusted label can be reliably supervised learning, but the processing of the untrusted label is still a key challenge. Simply discarding the sample image corresponding to the untrusted label or uniformly reducing its influence may cause information loss or unsatisfactory optimization results. Therefore, when prompt learning is performed, it is necessary to retain potentially useful learning signals from the sample image corresponding to the untrusted label, while preventing the introduction of harmful gradient directions.

[0121] Correspondingly, in the embodiments of the present application, a trust alignment gradient projection is introduced. Through the gradient level denoising mechanism of the trust alignment gradient projection, the semantic supervision number from the sample corresponding to the trusted label is used to construct a trust alignment space in the average gradient direction of the trusted label, and the gradient of the untrusted label is projected into the trust alignment space, thereby optimizing the gradient of the untrusted label.

[0122] In some embodiments, the server can first determine the average gradient of the trusted label. Second, the server can determine the cosine similarity corresponding to the gradient of the untrusted label according to the average gradient of the trusted label. Third, the server can optimize the gradient of the untrusted label according to the cosine similarity corresponding to the gradient of the untrusted label. Finally, the server can optimize the visual language model using the optimized gradient of the untrusted label and the gradient of the trusted label.

[0123] For example, the image sample set corresponding to the trusted label can be , the image sample set corresponding to the untrusted label can be , and the average gradient of the trusted label can be as shown in formula (15). For each untrusted sample, its gradient can be calculated according to the corresponding soft label, as shown in formula (16). Subsequently, the cosine similarity corresponding to the gradient of the untrusted label is calculated by formula (17).

[0124] (15)

[0125] (16)

[0126] (17)

[0127] wherein, is the training loss, is the model parameter, is the average gradient of the trusted label. the cosine similarity of the i-th untrusted label, the cosine similarity of the i-th untrusted label.

[0128] In some embodiments, based on the gradient consistency principle, the gradient of the untrusted label can be conditionally adjusted. If the cosine similarity is less than zero, the server projects the gradient of the untrusted label into the trust alignment space corresponding to the average gradient of the trusted labels. Then, the server optimizes the gradient of the untrusted label by removing the component of the gradient of the untrusted label in the direction of the average gradient in the trust alignment space. If the cosine similarity is greater than or equal to zero, the server can directly use the gradient of the untrusted label as the optimized gradient of the untrusted label.

[0129] For example, if , the gradient of the untrusted label is retained as the optimized gradient of the untrusted label; if , the gradient of the untrusted label is projected into the trust alignment space corresponding to the average gradient of the trusted labels, and the component of the gradient in the direction of is suppressed or removed to mitigate its adverse impact on model optimization. For example, the optimized gradient of the untrusted label can be as shown in equation (18). For example, the final optimization direction of the visual language model is determined by the optimized gradient of the untrusted label and the gradient of the trusted label, which can be as shown in equation (19).

[0130] (18)

[0131] (19)

[0132] wherein, is a gradient clipping coefficient for controlling the suppression intensity based on projection, for example, The value of may be 1.

[0133] In this application, a gradient consistency constraint is introduced in the optimization process of the visual language model, which suppresses harmful updates from untrusted labels by aligning the gradient direction with the supervision of trusted labels and mitigates convergence drift.

[0134] In some embodiments, the loss function of the visual language model can be a noise-aware supervised loss function, which can enhance the training robustness of untrusted samples.

[0135] For example, the loss function of the noise-aware supervision can be shown as formula (20). When q=0, the formula is simplified as the standard cross-entropy, and when q=1, the formula is linearly equivalent to the mean absolute error. For example, q can be set to 0.5 to balance between robustness and optimization efficiency, so that the loss function of the noise-aware supervision provides a stable pseudo-supervision signal, which helps to alleviate the adverse effects of untrusted labels in the training process.

[0136] (20)

[0137] wherein, is the loss function of the noise-aware supervision, q is a noise parameter, .

[0138] S204, using the optimized visual language model, identifying the to-be-identified image in the downstream task, and determining the category of the to-be-identified image.

[0139] In this step, after obtaining the optimized visual language model, the optimized visual language model can be used to identify the to-be-identified image in the downstream task, and determine the category of the to-be-identified image.

[0140] In some embodiments, the terminal can upload the to-be-identified image in the downstream task to the server, and after the server receives the to-be-identified image, the to-be-identified image can be input into the optimized visual language model, and the category recognition result output by the optimized visual language model can be obtained.

[0141] For example, the downstream task includes a case slide classification task, a hyperspectral object recognition task, and the like, and the embodiments of the present application do not limit this.

[0142] The method provided in the embodiments of the present application comprises the following steps: obtaining a noise sample data set of a visual language model, wherein the noise sample data set comprises a plurality of image samples and a plurality of sample labels corresponding to the image samples respectively; dividing the sample labels into trusted labels and untrusted labels according to a semantic trust boundary of the sample labels; performing prompt learning on the visual language model according to the image samples of the trusted labels and the image samples of the untrusted labels, so as to optimize the visual language model; and identifying an image to be identified in a downstream task by using the optimized visual language model, and determining the category of the image to be identified. The visual language model is a model that aligns image and text representations in a shared embedding space through pre-training. The prompt learning is used to adjust a context vector in the visual language model by using the gradient of the trusted labels and the gradient of the untrusted labels. The gradient of the untrusted labels is generated by projecting and optimizing the gradient of the trusted labels. The context vector is used to generate a text prompt of each category in the downstream task, so as to map the text prompt to a text representation. Since the trusted labels and the untrusted labels are distinguished by the semantic trust boundary of the sample labels, and the gradient of the untrusted labels is projected and optimized according to the gradient of the trusted labels in the prompt learning, the optimization drift caused by the untrusted labels is suppressed, and the accuracy of identifying the category of the image by using the visual language model is improved.

[0143] The following describes how to determine the semantic trust boundary of the sample labels. Figure 3 The flowchart of another method for identifying a multi-modal based on noise labels provided in the embodiments of the present application is shown in FIG. 3. Figure 3 The method for identifying a multi-modal based on noise labels comprises the following steps S301-S307.

[0144] S301, obtaining a noise sample data set of a visual language model, wherein the noise sample data set comprises a plurality of image samples and a plurality of sample labels corresponding to the image samples respectively.

[0145] S302, performing zero-shot prediction on the sample images of the sample labels by using the visual language model, to obtain a zero-shot prediction result.

[0146] S303, determining a prediction category set corresponding to the sample labels according to the zero-shot prediction result, wherein the prediction category set comprises K categories with the highest probabilities of the image samples corresponding to the sample labels obtained by the zero-shot prediction of the visual language model, and K is an integer greater than or equal to zero.

[0147] S304, determining a semantic trust boundary of the sample labels according to the prediction category set corresponding to the sample labels. The sample labels within the semantic trust boundary are within the prediction category set, and the sample labels outside the semantic trust boundary are not within the prediction category set.

[0148] S305, divide the sample labels into trusted labels and untrusted labels according to the semantic trust boundary of the sample labels.

[0149] S306, respectively prompt the visual language model according to the image samples of the trusted labels and the image samples of the untrusted labels to optimize the visual language model.

[0150] The visual language model is a model that aligns image and text representations in a shared embedding space through pre-training; the prompt learning is used to adjust the context vector in the visual language model through the gradient of the trusted label and the gradient of the untrusted label; the gradient of the untrusted label is generated after projection optimization according to the gradient of the trusted label; the context vector is used to generate a text prompt for each class in the downstream task to map the text prompt to the text representation.

[0151] S307, use the optimized visual language model to recognize the to-be-recognized image in the downstream task to determine the class of the to-be-recognized image.

[0152] In the embodiments of the application, a semantic trust boundary is introduced to distinguish reliable label regions, and trusted labels and untrusted labels are divided, the sensitivity of the model to label noise is enhanced by explicitly integrating the semantic prior of the visual language model, thereby improving the accuracy of dividing the trusted labels and the untrusted labels, and further improving the recognition accuracy.

[0153] The following describes how to project and optimize the gradient of the untrusted label according to the gradient of the trusted label. Figure 4 Another flowchart of a noise label based multi-modal recognition method provided by the embodiments of the application is shown in FIG. 4, which includes S401-S408. Figure 4

[0154] S401, obtain a noise sample dataset of a visual language model, and the noise sample dataset includes a plurality of image samples and a plurality of sample labels corresponding to the image samples respectively.

[0155] S402, divide the sample labels into trusted labels and untrusted labels according to the semantic trust boundary of the sample labels.

[0156] S403, respectively prompt the visual language model according to the image samples of the trusted labels and the image samples of the untrusted labels.

[0157] ​The visual language model is a model pre-trained to align image and text representations in a shared embedding space; the prompt learning is used to adjust a context vector in the visual language model by a gradient of trusted labels and a gradient of untrusted labels; the gradient of untrusted labels is generated after projection optimization according to the gradient of trusted labels; and the context vector is used to generate a text prompt for each category in a downstream task to map the text prompt to a text representation.

[0158] S404, determining an average gradient of the trusted labels.

[0159] S405, determining a cosine similarity corresponding to the gradient of untrusted labels according to the average gradient of the trusted labels.

[0160] S406, optimizing the gradient of untrusted labels according to the cosine similarity corresponding to the gradient of untrusted labels.

[0161] S407, optimizing the visual language model using the optimized gradient of untrusted labels and the gradient of trusted labels.

[0162] S408, identifying a to-be-identified image in a downstream task using the optimized visual language model to determine a category of the to-be-identified image.

[0163] The multi-modal recognition method based on noise labels provided by the embodiments of the present application generally achieves the best performance in all evaluation benchmarks and noise configurations. In severe noise conditions (especially when the label damage rate is as high as 75%), the multi-modal recognition method based on noise labels is significantly better than all competitive methods. For example, under the condition of 75% asymmetric noise, the performance of the multi-modal recognition method based on noise labels on the noise sample dataset (Flowers102) is 10.8% higher than that of the natural language prompt (NLPrompt) method, and the performance on the noise sample dataset (DTD) dataset is 13.21% higher than that of the NLPrompt method. In sharp contrast, the performance of the method in the related art (for example, the NLPrompt method) decreases sharply with the increase of the noise rate, and the accuracy rate is usually reduced to below 20%, which highlights the strong resistance to label noise accumulation.

[0164] In the embodiments of the present application, a trusted supervision space is constructed by aggregating the average gradient direction of trust labels, that is, the trust alignment space corresponding to the average gradient of trust labels. Then, the gradient of non-trust labels is projected into the trust alignment space corresponding to the average gradient of trust labels, and only the gradient component of the gradient of non-trust labels that is semantically aligned with the average gradient of trust labels is retained. The visual language model trained in this way does not discard or reduce the weight of the sample image corresponding to the non-trust label, thereby effectively suppressing semantic interference while maintaining effective supervision, thereby significantly enhancing the robustness of rapid tuning under label noise conditions, and thereby improving the accuracy of multi-modal recognition based on noisy labels.

[0165] The multi-modal recognition method based on noisy labels provided by the embodiments of the present application acquires a noisy sample data set of a visual language model, the noisy sample data set including a plurality of image samples and a plurality of sample labels corresponding to the plurality of image samples respectively; divides the sample labels into trust labels and non-trust labels according to a semantic trust boundary of the sample labels; performs prompt learning on the visual language model respectively according to the image samples of the trust labels and the image samples of the non-trust labels, to optimize the visual language model; and uses the optimized visual language model to recognize a to-be-recognized image in a downstream task and determine a category of the to-be-recognized image. The visual language model is a model that aligns image and text representations in a shared embedding space through pre-training. The prompt learning is used to adjust a context vector in the visual language model through the gradient of the trust labels and the gradient of the non-trust labels. The gradient of the non-trust labels is generated after projection optimization according to the gradient of the trust labels. The context vector is used to generate a text prompt of each category in the downstream task, to map the text prompt to a text representation. Since the trust labels and the non-trust labels are distinguished according to the semantic trust boundary of the sample labels, and the gradient of the non-trust labels is projected and optimized according to the gradient of the trust labels in the prompt learning, the optimization drift caused by the non-trust labels is suppressed, and the accuracy of recognizing the category of the image using the visual language model is improved.

[0166] It should be understood that, although each step in the flowchart involved in each of the above embodiments is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart involved in each of the above embodiments can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least part of other steps or steps or stages in other steps.

[0167] Based on the same inventive concept, the embodiments of the present application also provide a noise label based multi-modal recognition device for implementing the noise label based multi-modal recognition method described above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more noise label based multi-modal recognition device embodiments provided below can refer to the limitations of the noise label based multi-modal recognition method in the above, which will not be described here again.

[0168] In one exemplary embodiment, as shown in Figure 5 A noise label based multi-modal recognition device 500 is provided, comprising an acquisition module 501, a division module 502, a learning module 503 and an identification module 504, wherein:

[0169] The acquisition module 501 is configured to acquire a noise sample data set of a visual language model, wherein the noise sample data set comprises a plurality of image samples and sample labels corresponding to the plurality of image samples respectively.

[0170] The division module 502 is configured to divide the sample labels into trusted labels and untrusted labels according to a semantic trust boundary of the sample labels.

[0171] The learning module 503 is configured to respectively perform prompt learning on the visual language model according to the image samples of the trusted labels and the image samples of the untrusted labels, so as to optimize the visual language model.

[0172] The identification module 504 is configured to identify a to-be-identified image in a downstream task using the optimized visual language model, and determine a category of the to-be-identified image.

[0173] The visual language model is a model for aligning image and text representations in a shared embedding space through pre-training; the prompt learning is used to adjust a context vector in the visual language model through a gradient of the trusted labels and a gradient of the untrusted labels; the gradient of the untrusted labels is generated after projection optimization according to the gradient of the trusted labels; and the context vector is used to generate a text prompt of each category in the downstream task, so as to map the text prompt to the text representation.

[0174] In one embodiment, the division module 502 is further configured to perform zero-shot prediction on the sample images of the sample labels using the visual language model to obtain a zero-shot prediction result, and determine the semantic trust boundary of the sample labels according to the zero-shot prediction result.

[0175] In one of the embodiments, the dividing module 502 is further configured to determine, according to the zero-shot prediction result, a prediction class set corresponding to the sample label, the prediction class set including K classes with the highest probabilities of the image sample corresponding to the sample label being predicted by the zero-shot prediction of the visual language model, K being an integer greater than or equal to zero; determine a semantic trust boundary of the sample label according to the prediction class set corresponding to the sample label; wherein the sample label within the prediction class set is within the semantic trust boundary, and the sample label not within the prediction class set is outside the semantic trust boundary.

[0176] In one of the embodiments, the dividing module 502 is further configured to determine the precision and recall of the prediction class set corresponding to different K values; determine the probability index score of the prediction class set corresponding to different K values according to the precision and recall of the prediction class set corresponding to different K values; and determine the K value corresponding to the highest probability index score as the number of classes included in the prediction class set.

[0177] In one of the embodiments, the learning module 503 is further configured to determine the average gradient of the trust label; determine the cosine similarity of the gradient of the non-trust label according to the average gradient of the trust label; optimize the gradient of the non-trust label according to the cosine similarity of the gradient of the non-trust label; and optimize the visual language model using the optimized gradient of the non-trust label and the gradient of the trust label.

[0178] In one of the embodiments, the learning module 503 is further configured to, if the cosine similarity is less than zero, project the gradient of the non-trust label into a trust alignment space corresponding to the average gradient of the trust label; and optimize the gradient of the non-trust label by removing the component of the gradient of the non-trust label in the direction of the average gradient in the trust alignment space.

[0179] The above various modules in the multi-modal recognition device based on the noisy label can be realized by software, hardware, and combinations thereof, in whole or in part. The above various modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to the above various modules.

[0180] In one of the embodiments, a computer device is provided, which can be a server, and the internal structure diagram of the computer device can be as shown in Figure 6As shown in the figure. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through the system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store XX data. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with the terminal outside through the network connection. The computer program is executed by the processor to realize a multi-modal recognition method based on noise label.

[0181] Those skilled in the art can understand that, Figure 6 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or less components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0182] In one exemplary embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to realize the multi-modal recognition method based on noise label described above.

[0183] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by the processor to realize the multi-modal recognition method based on noise label described above.

[0184] In one embodiment, a computer program product is provided, including a computer program, and the computer program is executed by the processor to realize the multi-modal recognition method based on noise label described above.

[0185] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing relevant hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, the processes of the above-mentioned embodiment methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., but is not limited thereto. The processor involved in the embodiments provided in the present application can be a general processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but is not limited thereto.

[0186] The technical features of the above embodiments can be combined arbitrarily. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combinations of the technical features do not exist contradictory, it should be considered as the scope of the present application.

[0187] The above embodiments only express several implementation ways of the present application, and the description is specific and detailed, but it should not be understood as a limitation to the patent scope of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A multimodal recognition method based on noise tags, characterized in that, The method includes: Obtain a noisy sample dataset for a visual language model, wherein the noisy sample dataset includes multiple image samples and sample labels corresponding to the multiple image samples respectively; The visual language model is used to perform zero-shot prediction on the sample images of the sample labels to obtain zero-shot prediction results; Based on the zero-sample prediction results, the semantic trust boundary of the sample label is determined; Based on the semantic trust boundary of the sample labels, the sample labels are divided into trusted labels and untrusted labels; Based on the image samples of the trusted labels and the image samples of the untrusted labels, the visual language model is subjected to cue learning to optimize the visual language model. The optimized visual language model is used to identify the images to be identified in the downstream task and determine the category of the images to be identified. The visual language model is a model that aligns image and text representations within a shared embedding space through pre-training; the cue learning is used to adjust the context vector in the visual language model using the gradients of trusted labels and untrusted labels; the gradient of the untrusted label is generated by projection optimization based on the gradient of the trusted label; the context vector is used to generate text cue for each category in the downstream task, so as to map the text cue to the text representation; The optimization of the visual language model includes: Determine the average gradient of the trust label; Based on the average gradient of the trusted label, determine the cosine similarity corresponding to the gradient of the untrusted label; The gradient of the untrusted label is optimized based on the cosine similarity corresponding to the gradient of the untrusted label. The visual language model is optimized using the optimized gradients of the untrusted labels and the gradients of the trusted labels.

2. The method according to claim 1, characterized in that, Determining the semantic trust boundary of the sample label based on the zero-sample prediction result includes: Based on the zero-shot prediction results, a set of predicted categories corresponding to the sample label is determined. The set of predicted categories includes the K categories with the highest probability obtained by the zero-shot prediction of the image sample corresponding to the sample label through the visual language model, where K is an integer greater than or equal to zero. Based on the predicted category set corresponding to the sample label, the semantic trust boundary of the sample label is determined; wherein, sample labels within the predicted category set are within the semantic trust boundary, and sample labels not within the predicted category set are outside the semantic trust boundary.

3. The method according to claim 2, characterized in that, Before determining the set of predicted categories corresponding to the sample label based on the zero-sample prediction result, the method further includes: Determine the precision and recall corresponding to the predicted category sets with different values ​​of K; Based on the precision and recall of the predicted category sets with different K values, determine the probability index score corresponding to the predicted category sets with different K values; The value of K corresponding to the highest probability index score is determined as the number of categories included in the predicted category set.

4. The method according to claim 1, characterized in that, The step of optimizing the gradient of the untrusted label based on the cosine similarity corresponding to the gradient of the untrusted label includes: If the cosine similarity is less than zero, then the gradient of the untrusted label is projected into the trust alignment space corresponding to the average gradient of the trusted label; The gradient of the untrusted label is optimized by removing the component of the gradient of the untrusted label in the direction of the average gradient in the trust alignment space.

5. A multimodal recognition device based on noise tags, characterized in that, The device includes: The acquisition module is used to acquire a noise sample dataset of a visual language model, wherein the noise sample dataset includes multiple image samples and sample labels corresponding to the multiple image samples respectively; The segmentation module is used to perform zero-shot prediction on the sample images of the sample labels using a visual language model to obtain zero-shot prediction results; determine the semantic trust boundary of the sample labels based on the zero-shot prediction results; and divide the sample labels into trusted labels and untrusted labels based on the semantic trust boundary of the sample labels. The learning module is used to perform cue learning on the visual language model based on image samples of the trusted labels and image samples of the untrusted labels, respectively, so as to optimize the visual language model; The recognition module is used to identify the image to be recognized in the downstream task using an optimized visual language model, and to determine the category of the image to be recognized. The visual language model is a model that aligns image and text representations within a shared embedding space through pre-training; the cue learning is used to adjust the context vector in the visual language model using the gradients of trusted labels and untrusted labels; the gradient of the untrusted label is generated by projection optimization based on the gradient of the trusted label; the context vector is used to generate text cue for each category in the downstream task, so as to map the text cue to the text representation; The learning module is further configured to determine the average gradient of trusted labels; determine the cosine similarity corresponding to the gradient of untrusted labels based on the average gradient of trusted labels; optimize the gradient of untrusted labels based on the cosine similarity corresponding to the gradient of untrusted labels; and optimize the visual language model using the optimized gradient of untrusted labels and the gradient of trusted labels.

6. The apparatus according to claim 5, characterized in that, The partitioning module is further configured to determine the set of predicted categories corresponding to the sample labels based on the zero-shot prediction results. The set of predicted categories includes the K categories with the highest probabilities obtained by the zero-shot prediction of the image samples corresponding to the sample labels through the visual language model, where K is an integer greater than or equal to zero. Based on the set of predicted categories corresponding to the sample labels, the semantic trust boundary of the sample labels is determined. Sample labels within the set of predicted categories are within the semantic trust boundary, while sample labels not within the set of predicted categories are outside the semantic trust boundary.

7. The apparatus according to claim 6, characterized in that, The segmentation module is also used to determine the precision and recall corresponding to the predicted category sets with different K values; based on the precision and recall corresponding to the predicted category sets with different K values, determine the probability index score corresponding to the predicted category sets with different K values; and determine the K value corresponding to the highest probability index score as the number of categories contained in the predicted category set.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images