Methods, apparatus, equipment, storage media, and program products for labeling open set data

Abstract

This application discloses a method, apparatus, device, storage medium, and program product for labeling open set data, relating to the field of artificial intelligence technology. The method includes: acquiring open set data containing multiple samples; identifying N predicted known category samples from the open set data using a recognizer; and selecting predicted known category samples with satisfactory confidence levels from the N predicted known category samples as training samples for a classifier. This application uses a recognizer to initially identify samples in the open set data to obtain predicted known category samples, and then performs a secondary screening based on confidence levels to obtain training samples for training the classifier. This application provides a sample identification and labeling method applicable to open set data, and this method utilizes a recognizer for initial identification, followed by a secondary screening based on confidence levels, fully ensuring the accuracy of the known category samples selected from the open set data.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, storage medium, and program product for labeling open set data. Background Technology

[0002] When performing subsequent training or analysis on the target data in the dataset, it is necessary to label the target data from the dataset.

[0003] In a dataset containing only samples of known categories, a recognizer can identify the category of each sample in the dataset, and then select the samples belonging to the target category for subsequent training or analysis.

[0004] However, for datasets containing both known and unknown class samples, the above method cannot accurately distinguish between known and unknown class samples. Summary of the Invention

[0005] This application provides a method, apparatus, device, storage medium, and program product for labeling open set data. The technical solution is as follows:

[0006] According to one aspect of the embodiments of this application, a method for labeling open set data is provided, the method comprising:

[0007] Obtain open set data containing multiple samples, wherein the multiple samples include at least one known category sample and at least one unknown category sample; wherein, the known category sample refers to the sample whose true category belongs to K known categories, and the unknown category sample refers to the sample whose true category does not belong to the K known categories, where K is a positive integer;

[0008] N predicted known category samples are obtained from the open set data by a recognizer; wherein, the recognizer is used to identify the sample as a known category sample or an unknown category sample, the predicted known category sample refers to the sample identified by the recognizer as a known category sample, and N is a positive integer;

[0009] From the N predicted known category samples, the predicted known category samples that meet the confidence condition are selected as the training samples of the classifier; wherein, the classifier is used to classify the K known categories.

[0010] According to one aspect of the embodiments of this application, an audio content publishing apparatus is provided, the apparatus comprising:

[0011] The sample acquisition module is used to acquire open set data containing multiple samples, wherein the multiple samples include at least one known category sample and at least one unknown category sample; wherein, the known category sample refers to the sample whose true category belongs to K known categories, and the unknown category sample refers to the sample whose true category does not belong to the K known categories, where K is a positive integer;

[0012] A sample identification module is used to identify N predicted known category samples from the open set data using a recognizer; wherein, the recognizer is used to identify the sample as a known category sample or an unknown category sample, the predicted known category sample refers to the sample identified by the recognizer as a known category sample, and N is a positive integer;

[0013] The sample selection module is used to select samples of the predicted known categories that meet the confidence conditions from the N predicted known category samples as training samples for the classifier; wherein the classifier is used to classify the K known categories.

[0014] According to one aspect of the embodiments of this application, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement the above-described open set data annotation method.

[0015] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, code set, or instruction set is stored in the storage medium, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the above-described method for annotating open set data.

[0016] According to one aspect of the embodiments of this application, a computer program product is provided, the computer program product including computer instructions stored in a computer-readable storage medium, and a processor reading from the computer-readable storage medium and executing the computer instructions to implement the above-described open set data annotation method.

[0017] The technical solutions provided in this application embodiment may have the following beneficial effects:

[0018] This application addresses the problem that conventional annotation methods cannot be applied to open-set data, providing a method for identifying and annotating samples of known categories from open-set data. Furthermore, the method utilizes an identifier for initial identification followed by a confidence-based secondary selection, ensuring the accuracy of the known category samples selected from the open-set data. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the implementation environment of a solution provided in one embodiment of this application;

[0020] Figure 2 This is a flowchart of an open set data annotation method provided in one embodiment of this application;

[0021] Figure 3 This is a schematic diagram of an open set data annotation method provided in one embodiment of this application;

[0022] Figure 4 This is a flowchart of an open set data annotation method provided in another embodiment of this application;

[0023] Figure 5 This is an experimental result diagram provided in one embodiment of this application;

[0024] Figure 6 This is an experimental result diagram provided in another embodiment of this application;

[0025] Figure 7 This is an experimental result diagram provided in another embodiment of this application;

[0026] Figure 8 This is a block diagram of an open set data annotation apparatus provided in one embodiment of this application;

[0027] Figure 9 This is a block diagram of an open set data annotation apparatus provided in another embodiment of this application;

[0028] Figure 10 This is a schematic diagram of the structure of a computer device provided in one embodiment of this application. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0030] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0031] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0032] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0033] This application relates to the field of machine learning in artificial intelligence technology. The technical solution of this application will be described and explained through several embodiments below.

[0034] Before describing the embodiments of this application, some terms involved in this application will be explained.

[0035] 1. Active Learning (AL): This is a training method that attempts to train a better performing model by selectively labeling less data.

[0036] 2. Open set data: is a set of samples that includes samples of known categories and samples of unknown categories.

[0037] 3. Known category sample: The category is any one of the at least one categories that the recognizer can identify.

[0038] 4. Unknown category sample: A sample whose category is not any of the at least one category that the recognizer can recognize.

[0039] 5. Open-set annotation (OSA): refers to labeling samples in open-set data with their categories.

[0040] 6. Open-set Recognition (OSR): refers to the identification of categories for samples in open-set data.

[0041] 7. Gaussian Mixture Model (GMM): This is a probability model that can be used to represent a population distribution containing several sub-distributions. A Gaussian Mixture Model can be viewed as a model composed of several single Gaussian models, and these sub-models are the latent variables of the mixture model.

[0042] Please refer to Figure 1 The diagram illustrates an implementation environment for a solution provided in one embodiment of this application. This implementation environment can be implemented as an open-set data annotation system. The implementation environment may include a model training device 10 and a model usage device 20.

[0043] The model training device 10 can be an electronic device such as a computer, server, or intelligent robot, or other electronic devices with strong computing power. The model training device 10 includes a recognizer 30 and a classifier 40. The recognizer 30 is used to label open-set data, obtaining labeled known category samples, and then training the classifier 40 using these labeled known category samples. In this embodiment, labeling open-set data refers to labeling the samples in the open-set data with categories. The labeling of open-set data can also be used to train the classifier 40. By labeling the samples in the open-set data with categories, corresponding labeling results are obtained to obtain training samples for the classifier 40, which are then used to train the classifier 40. The recognizer 30 in the model training device 10 is trained through active learning. The recognizer 30 labels the samples in the open-set data, obtains labeled known category samples through manual review, and trains the recognizer 30 based on these labeled known category samples. A trained recognizer is obtained through multiple training iterations. The classifier 40 is trained using a supervised learning method. The sample in the open set data is labeled by the recognizer 30 to obtain the labeled known category samples, and the classifier 40 is trained based on the labeled known category samples.

[0044] The trained classifier 40 can be deployed on the model-using device 20 to identify the category of the target sample. The model-using device 20 can be a terminal device such as a mobile phone, computer, smart TV, multimedia playback device, wearable device, medical device, or a server; this application does not limit this.

[0045] In some embodiments, such as Figure 1 As shown, the recognizer 30 obtains known category samples 32 by labeling and extracting known category samples from the open set data 31, where known category samples 32 are images labeled with thick outlines in the open set data 31. The known category samples 32 are then labeled to obtain training samples, which are then used to train the classifier 40.

[0046] The following method embodiments will detail the usage and training process of the recognizer 30, as well as the training process of the classifier 40.

[0047] Please refer to Figure 2 The diagram illustrates a flowchart of an open set data annotation method provided in one embodiment of this application. The entity executing this method may be... Figure 1 The model training device 10 shown can be used to perform the various steps. The method may include at least one of the following steps (210-230):

[0048] Step 210: Obtain open set data containing multiple samples.

[0049] The open set data contains multiple samples, including at least one sample of a known class and at least one sample of an unknown class. The sample of a known class is a sample whose true class belongs to one of the K known classes, and the sample of an unknown class is a sample whose true class does not belong to one of the K known classes, where K is a positive integer.

[0050] For example, let's take image samples from an open set of data. Assuming the K known categories include football, basketball, and volleyball, then the known category samples in the open set are samples of category football, basketball, and volleyball. Unknown category samples are those whose category is not any of football, basketball, or volleyball. For example, image samples of categories like swimming, houses, or rainbows would be considered unknown category samples.

[0051] It should be noted that the model training device acquires open set data containing multiple samples. Each sample in this open set data is not labeled as known or unknown. The method provided in this embodiment is needed to distinguish between known category samples and unknown category samples from this open set data.

[0052] Step 220: N predicted known category samples are obtained from the open set data by the recognizer, where N is a positive integer.

[0053] Optionally, the recognizer is used to identify whether a sample is a known category sample or an unknown category sample. The recognizer can be a neural network model, and the network structure of the recognizer is not limited in this embodiment. In some embodiments, the recognizer is used to identify known category samples and unknown category samples. In other embodiments, the recognizer is used to identify known category samples corresponding to K known categories, and unknown category samples.

[0054] Predicted known-class samples refer to samples that the recognizer identifies as belonging to a known class. In other words, predicted known-class samples are those selected by the recognizer from the open set of data that it considers to be of a known class. However, since the recognizer's results are not always accurate, predicted known-class samples could be genuinely known-class samples, or they could be samples of unknown classes.

[0055] Optionally, the recognizer obtains known category samples and unknown category samples based on the prediction. The known category samples and unknown category samples are the prediction results obtained by the recognizer, and their true categories may differ from the prediction results. For example, if the model training device predicts the known category samples to be basketball and football, and the true category of a sample is football, but the recognizer identifies the sample and obtains a predicted category of football, then the recognizer labels the sample as a known category sample, and the recognizer's prediction is correct. Optionally, if the recognizer identifies the sample and obtains a predicted category of basketball, then the recognizer labels the sample as a known category sample, but the recognizer's prediction is incorrect. The solution for this situation is shown in the following embodiments. Optionally, if the recognizer identifies the sample and obtains a predicted category of volleyball, then the recognizer labels the sample as an unknown category sample, and the recognizer's prediction is incorrect. Optionally, the recognizer may also identify an unknown category sample as a known category sample, which will not be elaborated further. In summary, the recognizer has the problem of recognition errors. Therefore, it is necessary to train the recognizer to improve its recognition accuracy. The specific training method is described in the examples below.

[0056] Optionally, the recognizer obtains predicted known-class samples and predicted unknown-class samples from the open set data. The sum of the number of predicted known-class samples and predicted unknown-class samples equals the number of samples in the open set data. Here, predicted unknown-class samples refer to samples that the recognizer identifies as belonging to the unknown class. Since predicted unknown-class samples are not used in the training process of either the recognizer or the classifier, only predicted known-class samples need to be selected for subsequent processes.

[0057] In some embodiments, the predicted known category samples obtained by the recognizer are sampled to obtain N predicted known category samples. For example, the open set data includes 1,000,000 samples. Through the recognizer's prediction, 100,000 predicted known category samples and 900,000 predicted unknown category samples are obtained. Then, through random sampling, 1,000 predicted known category samples are randomly sampled from the 100,000 predicted known category samples for the next step. In this example, the N predicted known category samples in step 220 can be understood as the aforementioned 1,000 predicted known category samples.

[0058] In some embodiments, such as Figure 3 As shown, Figure 3 An exemplary diagram illustrates how a recognizer identifies samples in open-set data. Optionally, Figure 3 It consists of the recognition process of recognizer 310, the training process of recognizer 30, and the training process of classifier 40. The recognition process of recognizer 30 is used to label and select known category samples from the open set data for recognitionr 30 and classifier 40 to train. Figure 3 The open set data is input into the recognizer 30 to obtain N samples with known predicted categories.

[0059] Step 230: Select the predicted known category samples that meet the confidence condition from the N predicted known category samples as training samples for the classifier; wherein, the classifier is used to classify K known categories.

[0060] The confidence level requirement is used to perform secondary prediction on the predicted known category samples identified by the recognizer. Through secondary filtering, it aims to distinguish as many unknown category samples as possible from the predicted known category samples, thereby obtaining the true known category samples from the predicted known category samples and improving the accuracy of the final selected known category samples. Optionally, such as... Figure 3 As shown, the Gaussian Mixture Model (GMM) performs a secondary filtering on the predicted known class samples obtained by the classifier 30, resulting in filtered predicted known class samples. Optionally, the filtered predicted known class samples can be used as training samples for training the classifier.

[0061] This application uses a recognizer to initially identify samples in open-set data to obtain samples with predicted known categories. Then, it performs a secondary screening of these samples based on confidence level to obtain training samples for training the classifier. This application solves the problem that ordinary annotation methods cannot be applied to open-set data, providing a method for identifying and labeling samples with known categories from open-set data. Furthermore, this method utilizes a recognizer for initial identification and then performs a secondary screening based on confidence level, fully ensuring the accuracy of the known category samples selected from the open-set data.

[0062] Please refer to Figure 4 This illustrates a flowchart of an open set data annotation method provided in another embodiment of this application. The entity executing this method may be... Figure 1 The model training device 10 shown can be used to perform the various steps. The method may include at least one of the following steps (410-470):

[0063] Step 410: Obtain an open set of data containing multiple samples, including at least one known category sample and at least one unknown category sample; wherein, a known category sample refers to a sample whose true category belongs to K known categories, and an unknown category sample refers to a sample whose true category does not belong to K known categories, where K is a positive integer.

[0064] For a description of step 410, please refer to the above embodiment; it will not be repeated here.

[0065] Step 420: For the target sample in the open set data, input the target sample into the recognizer.

[0066] The target sample can be any sample in the open set of data. The target sample may be a sample with a known class or a sample with an unknown class.

[0067] The target sample is input into the recognizer, which then predicts the category of the target sample. For example... Figure 3 As shown, Figure 3 The sample data in the open set is input into the recognizer 30, and the recognizer 30 performs category prediction on the open set data.

[0068] Step 430: Obtain K+1 activation values ​​corresponding to the target sample through the recognizer; wherein, the K+1 activation values ​​correspond one-to-one with the K+1 categories, and the K+1 categories include unknown categories and K known categories.

[0069] The recognizer obtains K+1 activation values ​​for the target sample based on K+1 categories. These K+1 categories include K known categories and 1 unknown category. Optionally, the K known categories are the first K categories, and the unknown category is the (K+1)th category. The recognizer identifies each category of the target sample separately, obtaining the corresponding activation value for each category. For example, if the model training device sets K to 2, and the two known categories are football and basketball, the recognizer will identify the categories of the target sample, obtaining activation values ​​for three categories: football, basketball, and the unknown category. For instance, the activation value for football would be 1, for basketball 0.3, and for the unknown category 0.2.

[0070] Optionally, for any one of the K+1 target categories mentioned above, the activation value corresponding to that target category is used to characterize the probability that the target sample belongs to that target category. Optionally, the larger the activation value corresponding to the target category, the greater the probability that the target sample belongs to that target category. For example, the activation value corresponding to football is 1, the activation value corresponding to basketball is 0.3, and the activation value corresponding to the unknown category is 0.2. Among them, the activation value corresponding to football is the largest, so the probability that the target sample belongs to football is the highest. Based on this, the recognizer can consider the target sample to be a sample of a predicted known category, and its category is football. For another example, the activation values ​​corresponding to the target samples are as follows: the activation value corresponding to football is 0, the activation value corresponding to basketball is 0.3, and the activation value corresponding to the unknown sample is 0.9. Then the recognizer predicts that the target sample is most likely to be of the unknown category and considers the target sample to be a sample of a predicted unknown category.

[0071] Step 440: If K+1 activation values ​​meet the conditions, then the target sample is determined to be a known category sample.

[0072] The above conditions refer to the criteria used to determine whether a sample is a known category sample or an unknown category sample based on the activation value corresponding to the sample.

[0073] Optionally, step 440 includes: determining the maximum activation value among the K+1 activation values; if the category corresponding to the maximum activation value belongs to one of the K known categories, then the target sample is determined to be a known category sample.

[0074] The maximum activation value is the largest among K+1 activation values. The category of the sample is determined based on the category corresponding to the maximum activation value. If the category corresponding to the maximum activation value is any one of the K known categories, then the target sample is a known category sample; if the category corresponding to the maximum activation value is the (K+1)th category, i.e., the target category is unknown, then the target sample is an unknown category sample. For example, if the activation values ​​of each target category in the target sample are as follows: the activation value for soccer is 1, the activation value for basketball is 0.3, and the activation value for the unknown category is 0.2, then the maximum activation value of the target sample is 1, and its corresponding category is soccer. In this case, if soccer is any one of the K categories, then the target sample is a known category sample.

[0075] By calculating the activation values ​​corresponding to each category of the target sample, the category of the target sample is determined, and a target sample with a more accurate category is obtained, which provides a basis for the calculation of the probability value of the target sample in the following text.

[0076] Step 450: Select N samples from the open set data that are determined to be samples of the known class to obtain N predicted samples of the known class.

[0077] From the predicted known category samples and predicted unknown category samples obtained by the recognizer, select N predicted known category samples. Optionally, the N predicted known category samples are all or part of the predicted known category samples obtained by the recognizer.

[0078] Step 460: Use GMM to model the distribution of the maximum activation values ​​corresponding to the N predicted known class samples, and obtain the probability values ​​corresponding to the N predicted known class samples respectively.

[0079] The probability value is obtained by processing the activation values ​​corresponding to each category in the predicted known category samples using a Gaussian Mixture Model (GMM). The probability value characterizes the confidence that a predicted known category sample belongs to a known category; that is, the confidence of the recognizer's prediction of the target sample. The higher the probability value of the target sample, the higher the confidence of the prediction result, meaning the more accurate the prediction result. For example, given two samples A and B, where sample A has a probability value of 0.8 and sample B has a probability value of 0.7, sample A has higher confidence than sample B, and the recognizer's prediction result for sample A is more accurate than its prediction result for sample B.

[0080] Optionally, step 460 includes the following steps (1-3):

[0081] 1. For a target known category among K known categories, select M predicted known category samples that belong to the target known category from N predicted known category samples, where M is a positive integer less than or equal to N.

[0082] M samples of predicted known categories are randomly sampled from the N predicted known category samples obtained by the recognizer. Optionally, the first M samples from the predicted known category samples obtained by the recognizer are selected as the M predicted known category samples. Optionally, the recognizer stops predicting after it has predicted the M predicted known category samples. This application does not limit the method of obtaining the M predicted known category samples.

[0083] In some embodiments, such as Figure 3 As shown, Figure 3 The predicted known category samples in the above M predicted known category samples are obtained from the N predicted known category samples obtained by the recognizer 30 through random sampling.

[0084] 2. For each of the M predicted known category samples, obtain the activation value corresponding to the target known category from the K+1 activation values ​​corresponding to the predicted known category sample, and obtain M activation values.

[0085] The activation values ​​of the M predicted known category samples obtained above are determined by selecting the largest activation value from the activation values ​​of each category of any predicted known category sample among the M predicted known category samples. The category corresponding to this maximum activation value is then determined. The M activation values ​​corresponding to the M predicted known category samples are obtained using the above method.

[0086] 3. Use GMM to model the M activation values ​​to obtain the probability values ​​corresponding to the M predicted known class samples.

[0087] Here, the probability value corresponding to the i-th predicted known category sample among the M predicted known category samples is used to characterize the confidence that the i-th predicted known category sample belongs to the target known category, where i is a positive integer less than or equal to M.

[0088] Based on the Gaussian Mixture Model (GMM), the maximum activation values ​​of M predicted known-class samples are modeled to obtain the probability values ​​corresponding to the M predicted known-class samples. Each probability value is used to characterize the confidence level of its corresponding predicted known-class sample, that is, the confidence level that the predicted known-class sample is indeed a known-class sample.

[0089] By using the Gaussian mixture model and the activation values ​​of samples of the same class, the probability value of samples of that class is determined, which lays the foundation for predicting known class samples based on the magnitude of the probability value to obtain the predicted known class samples that meet the target conditions in the following text.

[0090] Step 470: From N predicted known category samples, obtain the training samples for the classifier based on the predicted known category samples whose probability values ​​satisfy the target conditions.

[0091] The target condition is used to further filter the predicted known class samples to obtain predicted known class samples whose probability values ​​satisfy the target condition. Optionally, the target condition can be a threshold, selecting predicted known class samples with probability values ​​greater than or equal to the threshold as training samples for the classifier. For example, if the threshold is set to 0.7, and there are three predicted known class samples with probability values ​​of 0.8 for sample A, 0.7 for sample B, and 0.6 for sample C, then samples A and B are selected as training samples for the classifier. Optionally, the target condition can be a fixed number of training samples. For example, if the fixed number of training samples is set to 100, then the top 100 probability values ​​are selected as training samples based on their magnitude.

[0092] Optionally, step 470 includes the following steps (471-472):

[0093] Step 471: Obtain the labeled category corresponding to the predicted known category sample whose probability value satisfies the target condition. The labeled category is one of the K+1 categories.

[0094] Optionally, before obtaining the training samples mentioned above, the predicted known category samples whose probability values ​​satisfy the target condition are first labeled with a category, where the labeled category is any one of the K+1 categories. Optionally, the labeled categories of the predicted known category samples whose probability values ​​satisfy the target condition can be labeled manually. Optionally, the labeled categories of the predicted known category samples whose probability values ​​satisfy the target condition can be labeled using active learning, resulting in labeled known category samples and labeled unknown category samples, where labeled known category samples are labeled known category samples and labeled unknown category samples are labeled unknown category samples.

[0095] Optionally, after step 471, the following steps (AB) are also included:

[0096] A. Calculate the training loss of the recognizer based on the labeled category corresponding to the predicted known category sample and the K+1 activation values ​​corresponding to the predicted known category sample obtained by the recognizer. Here, the predicted category is one of the K+1 categories.

[0097] Based on the labeled categories of any sample from the known-category and unknown-category samples, determine the activation values ​​corresponding to its K+1 categories. Then, compare and calculate the activation values ​​of the K+1 categories from the known-category or unknown-category samples corresponding to the predicted known-category samples with the activation values ​​of the predicted known-category samples to obtain the training loss of the recognizer. Based on this training loss, the recognizer is trained.

[0098] Optionally, based on the labeled category corresponding to the predicted known category sample and the K+1 activation values ​​corresponding to the predicted known category sample obtained by the recognizer, the training loss of the recognizer is calculated by the following steps (ab):

[0099] a. Based on the predicted known category samples corresponding to the labeled categories, determine the K+1 label activation values ​​corresponding to the predicted known category samples; wherein, the label activation value corresponding to the labeled category is the first value, and the label activation values ​​corresponding to the other categories in the K+1 categories besides the labeled category are the second values, and the first value and the second value are different;

[0100] Optionally, the annotation activation value of the labeled known-category sample or the labeled unknown-category sample is determined based on the category of the labeled known-category sample corresponding to the predicted known-category sample. The annotation activation value corresponding to the category of the labeled known-category sample or the labeled unknown-category sample is a first value, and the annotation activation value for other categories is a second value. For example, if the known category of a sample is "football," then the annotation activation value for "football" is 1 (the first value), and the annotation activation value for other categories is 0 (the second value). Optionally, if a sample is labeled with an unknown category, then the annotation activation value corresponding to the unknown category is 1, and the annotation activation value for other categories is 0. This application does not limit the values ​​of the first and second values.

[0101] b. Calculate the training loss of the recognizer based on the K+1 labeled activation values ​​corresponding to the predicted known class samples, the K+1 activation values ​​corresponding to the predicted known class samples obtained by the recognizer, and the temperature coefficient; where the temperature coefficient is used to adjust the sharpness of the activation value distribution.

[0102] Based on the first and second values ​​obtained above, and the activation values ​​of each category of the corresponding predicted known category samples, the training loss of the recognizer is calculated. This training loss is obtained by the difference between the first and second values ​​and the activation values ​​of each category of the corresponding predicted known category samples. Optionally, the training loss can also be adjusted using the temperature coefficient of the loss function. By adjusting the temperature coefficient, i.e., adjusting the coefficient of the loss function, a sharper activation value can be obtained, resulting in a training loss that is more effective at training the recognizer. The loss function is the function used to calculate the training loss of the recognizer. For example, by reducing the temperature coefficient of the loss function, a sharper distribution loss function can be obtained. The sharpness of the loss function refers to the size of the difference between the maximum and minimum values ​​of the function. The larger the difference between the maximum and minimum values, the sharper the loss function, and the better the training effect of the obtained training loss.

[0103] In some embodiments, such as Figure 3 As shown, the recognizer 30 is trained using a temperature system 330 and labeled samples of known and unknown categories.

[0104] By manually labeling samples of the predicted known category, labeled samples of the known category and labeled samples of the unknown category are obtained. The activation values ​​of the labeled samples of the known category and labeled samples of the unknown category are then labeled using a first value and a second value, so that the trained recognizer can better identify the category of the sample.

[0105] B. The parameters of the recognizer are adjusted based on the training loss of the recognizer to obtain the updated recognizer; wherein, the updated recognizer is used to identify new samples of the predicted known class from the open set data.

[0106] The parameters of the recognizer are adjusted using the training loss obtained above, so that the detection result obtained by the recognizer after detecting the predicted known category sample is the same as the labeled category corresponding to the predicted known category sample. Optionally, the activation value obtained by the recognizer after detecting the predicted known category sample can be approximately close to the first and second values ​​corresponding to the same predicted known category sample.

[0107] By training the aforementioned recognizer multiple times, a multi-trained recognizer is obtained. Compared to the original recognizer, the multi-trained recognizer can obtain more accurate recognition results, reducing the recognition errors that may occur in subsequent manual annotation.

[0108] like Figure 3 As shown, Figure 3 The activation values ​​for the labeled known category samples and labeled unknown category samples are the first and second values ​​mentioned above.

[0109] Step 472: Use the predicted known category samples whose labeled categories belong to K known categories as training samples for the classifier.

[0110] The labeled samples with known categories obtained above are used as training samples for the classifier.

[0111] Optionally, the model training device also needs to pre-train the recognizer. The specific steps are as follows: obtain the pre-training dataset of the recognizer, which includes at least one pre-training sample with a class label, where the class label is an unknown class and one of K known classes; train the recognizer using the pre-training dataset to obtain the pre-trained recognizer; wherein, the pre-trained recognizer is used to predict known class samples from the open set data.

[0112] Obtain a pre-training dataset, where samples are labeled with class categories. A labeled category refers to a sample whose category is known. For example, samples in the pre-training dataset might have known category labels such as soccer, basketball, or volleyball, or be unknown. Train the recognizer using the pre-training dataset. Calculate the training loss based on the predicted categories of the samples in the pre-training dataset and their actual categories. Then, pre-train the recognizer using this training loss to obtain the pre-trained recognizer.

[0113] Optionally, for labeled samples of known categories, a K-class classifier is trained by minimizing the standard cross-entropy loss, where the number of categories the classifier can recognize is the same as the number of known categories the recognizer can recognize. Optionally, the number of categories the classifier can recognize is less than the number of known categories the recognizer can recognize. This application does not limit the number of categories the classifier can recognize. The specific formula is as follows:

[0114]

[0115] Among them, (x i ,y i )∈D L D L These are the labeled samples of known categories, θ C These are the training parameters for a K-class classifier, n L It is the current set D L Size.

[0116] By pre-training the recognizer, it gains initial recognition capabilities, enabling it to initially identify categories, thus accelerating the training process of the recognizer.

[0117] This embodiment identifies the category of samples in open-set data using a recognizer, and then selects a portion of the predicted known category samples obtained by the recognizer through random sampling. Next, the probability values ​​of the predicted known category samples are calculated using activation values ​​and a Gaussian mixture model. Based on these probability values, samples meeting certain conditions are selected for labeling. These samples are then manually reviewed and labeled, resulting in labeled known category samples and labeled unknown category samples. Finally, the labeled known category samples are input into a classifier model for training, and the labeled unknown category samples are input into the recognizer for training. On one hand, the more accurate labeled known and unknown category samples obtained through manual review make the training of the recognizer and classifier models more effective. On the other hand, calculating the probability values ​​using a Gaussian mixture model and the activation values ​​of the predicted known category samples, and then selecting the predicted known category samples corresponding to the probability values ​​that meet the conditions for manual review, reduces the manpower and time required for manual review and improves the training efficiency of the labeling model for open-set data.

[0118] The following is the training process for the recognizer, with specific steps as follows:

[0119] The recognizer can identify K known categories and the (K+1)th unknown category. For manually labeled samples of known and unknown categories, taking sample X as an example, sample X is encoded using one-hot encoding (one-hot encoding uses an N-bit state register to encode N states), yielding the corresponding activation values. The activation value for the category corresponding to sample X is 1, while the activation values ​​for other categories are labeled as 0. The recognizer is optimized using a cross-entropy loss function with a temperature coefficient T, as shown in the following formula:

[0120]

[0121] in

[0122]

[0123] Among them, L D It is the cross-entropy loss function, L D (x,c) is the cross-entropy loss function of class C corresponding to sample X, a c The activation value of sample X for class C. The probability distribution of the cross-entropy loss function, calculated using the formula above, shows that known-class samples have larger activation values ​​in the first K known classes and smaller activation values ​​in the (K+1)th unknown class, while unknown-class samples exhibit the opposite phenomenon. Furthermore, reducing the temperature coefficient T of the cross-entropy loss function can increase the probability distribution. The range of values ​​for makes [the value range] ... More sharp. The specific formula is as follows:

[0124]

[0125] Among them, due to the decrease in the temperature coefficient T, The range of activation values ​​becomes larger, making it easier to distinguish between samples of known and unknown categories.

[0126] The following is the calculation process of a Gaussian mixture model: randomly sampling from N known-class predicted samples to obtain M known-class predicted samples, and then extracting the known-class predicted samples whose probability values ​​satisfy the target condition from the M known-class predicted samples. The specific steps are as follows:

[0127] During the training of the aforementioned recognizer, we discovered that activation values ​​can be used to determine whether a sample in the open set is a known-class or unknown-class sample. Specifically, the maximum activation value of an unknown-class sample is significantly different from the average activation value of a known-class sample. For each sample class C, the maximum activation value can be defined as follows:

[0128]

[0129] in, It is the maximum activation value of sample class C. Is seeking Maximum activation value, It is the activation value corresponding to sample category C of the labeled samples with known categories.

[0130] The predicted known class samples are input into a Gaussian mixture model, and the EM (Expectation-Maximum) algorithm is used to process them. We perform Gaussian mixture modeling and calculate the corresponding probability values:

[0131] W c =GMM(mav) c ,θ D )

[0132] Among them, W c θ represents the probability value corresponding to class C for each sample. D These are the parameters of the Gaussian mixture model. For each unlabeled sample x of class C... i The probability value w of its known class samples i It is a prior probability P(g|mav) i ), where g is the Gaussian component with the largest activation value. Then the probabilities of each category are merged and sorted.

[0133] W = sort(W 1 ∪W 2 ∪…∪W K )

[0134] Here, sort refers to arranging the probability values ​​in parentheses.

[0135] Next, the annotator labels the top b samples with the highest probability values. Optionally, a threshold is set to construct the query set for the annotator, as shown in the following formula:

[0136]

[0137] Among them, X query The query set mentioned above is used, and τ is the set threshold. After labeling the samples in the query set, the labeled samples of known categories and the labeled samples of unknown categories are updated for use in training the recognizer.

[0138] In some embodiments, the LFOSA active learning method of this scheme is compared with five other methods—Random, Uncertainty, OpenMax, Coreset, and BALD—in terms of sample selection recall, accuracy, and target model performance improvement. The publicly available datasets used are CIFAR10, CIFAR100, and Tiny-Imagenet. Figure 5 , Figure 6 and Figure 7 As shown, Figure 5 Examples are shown comparing the recall performance of various active learning methods in sample sampling. Figure 6 An example is shown comparing the accuracy performance of several different active learning methods in sample sampling. Figure 7 Examples are provided to compare the model performance improvements of various active learning methods based on sample sampling strategies. Figure 5 , Figure 6 and Figure 7 The experiments were conducted 9 times, yielding 5 different sets of experimental results.

[0139] In some embodiments, such as Figure 5 As shown, Figure 5 Curve 51 represents the sample sampling recall performance of the LFOSA active learning method in this application. In each graph, the horizontal axis represents the number of training iterations, and the vertical axis represents the sample sampling recall performance. It can be seen that regardless of the number of training iterations, the LFOSA active learning method in this application outperforms other methods.

[0140] Similarly, in some embodiments, such as Figure 6 As shown, Figure 6 Curve 61 represents the sample sampling accuracy performance of the LFOSA active learning method in this application. In each graph, the horizontal axis represents the number of training iterations, and the vertical axis represents the sample sampling accuracy performance. It can be seen that regardless of the number of training iterations, the LFOSA active learning method in this application outperforms other methods.

[0141] Similarly, in some embodiments, such as Figure 7 As shown, Figure 7 Curve 71 represents the model performance improvement of the sampling strategy in the LFOSA active learning method of this application. In each graph, the horizontal axis represents the number of training iterations, and the vertical axis represents the model performance improvement of the sampling strategy. It can be seen that before four training iterations, the LFOSA active learning method in this application shows comparable model performance improvements to other methods regarding the sampling strategy. However, after four training iterations, the LFOSA active learning method in this application demonstrates a more significant improvement in model performance than other methods regarding the sampling strategy.

[0142] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0143] Please refer to Figure 8 This diagram illustrates a block diagram of an open-set data annotation apparatus according to an embodiment of this application. The apparatus has the function of implementing the above-described open-set data annotation method; this function can be implemented in hardware or by hardware executing corresponding software. The apparatus can be the model training device described above, or it can be located within a model training device. The apparatus 800 may include: a sample acquisition module 810, a sample recognition module 820, and a sample selection module 830.

[0144] The sample acquisition module 810 is used to acquire open set data containing multiple samples, wherein the multiple samples include at least one known category sample and at least one unknown category sample; wherein, the known category sample refers to the sample whose true category belongs to K known categories, and the unknown category sample refers to the sample whose true category does not belong to the K known categories, where K is a positive integer.

[0145] The sample identification module 820 is used to identify N predicted known category samples from the open set data through a recognizer; wherein, the recognizer is used to identify the sample as a known category sample or an unknown category sample, the predicted known category sample refers to the sample identified by the recognizer as a known category sample, and N is a positive integer.

[0146] The sample selection module 830 is used to select samples of the predicted known categories that meet the confidence conditions from the N predicted known category samples as training samples for the classifier; wherein the classifier is used to classify the K known categories.

[0147] In an exemplary embodiment, such as Figure 9 As shown, the sample recognition module 820 may include: a sample input unit 821, an activation value acquisition unit 822, a category determination unit 833, and a predicted sample determination unit 834.

[0148] The sample input unit 821 is used to input the target sample in the open set data into the recognizer.

[0149] The activation value acquisition unit 822 is used to obtain K+1 activation values ​​corresponding to the target sample through the recognizer; wherein, the K+1 activation values ​​correspond one-to-one with K+1 categories, and the K+1 categories include unknown categories and the K known categories.

[0150] The category determination unit 833 is used to determine the target sample as the known category sample if the K+1 activation values ​​meet the conditions.

[0151] The prediction sample determination unit 834 is used to select N samples that are determined to be samples of the known category from the open set data to obtain the N predicted known category samples.

[0152] In an exemplary embodiment, the category determination unit 833 is configured to:

[0153] Determine the maximum activation value among the K+1 activation values;

[0154] If the category corresponding to the maximum activation value belongs to the K known categories, then the target sample is determined to be a sample of the known category.

[0155] In an exemplary embodiment, such as Figure 9 As shown, the sample selection module 830 may include a probability value acquisition unit 831 and a training sample acquisition unit 832.

[0156] The probability value acquisition unit 831 is used to model the maximum activation value distribution corresponding to the N predicted known category samples using a Gaussian mixture model (GMM) to obtain the probability values ​​corresponding to the N predicted known category samples respectively; wherein, the probability value is used to characterize the confidence that the predicted known category sample belongs to the known category.

[0157] The training sample acquisition unit 832 is used to obtain the training samples of the classifier from the N predicted known category samples based on the probability values ​​that satisfy the target conditions.

[0158] In some embodiments, the probability value acquisition unit 831 is configured to:

[0159] For the target known category among the K known categories, select M predicted known category samples belonging to the target known category from the N predicted known category samples, where M is a positive integer less than or equal to N;

[0160] For each of the M predicted known category samples, the activation value corresponding to the target known category is obtained from the K+1 activation values ​​corresponding to the predicted known category sample, thus obtaining M activation values;

[0161] The GMM is used to model the M activation values ​​to obtain the probability values ​​corresponding to the M predicted known category samples respectively; wherein, the probability value corresponding to the i-th predicted known category sample among the M predicted known category samples is used to characterize the confidence that the i-th predicted known category sample belongs to the target known category, and i is a positive integer less than or equal to M.

[0162] In some embodiments, the training sample acquisition unit 832 is configured to:

[0163] Obtain the labeled category corresponding to the predicted known category sample whose probability value satisfies the target condition, wherein the labeled category is one of the K+1 categories;

[0164] The predicted known category samples whose labeled categories belong to the K known categories are used as training samples for the classifier.

[0165] In some embodiments, the training sample acquisition unit 832 is further configured to:

[0166] Based on the labeled category corresponding to the predicted known category sample, and the K+1 activation values ​​corresponding to the predicted known category sample obtained by the recognizer, the training loss of the recognizer is calculated; wherein, the predicted category is one of the K+1 categories;

[0167] The parameters of the recognizer are adjusted based on the training loss of the recognizer to obtain an updated recognizer; wherein the updated recognizer is used to identify new samples of the predicted known class from the open set data.

[0168] In some embodiments, the training sample acquisition unit 832 is configured to:

[0169] Based on the labeled categories corresponding to the predicted known category samples, K+1 label activation values ​​corresponding to the predicted known category samples are determined; wherein, the label activation value corresponding to the labeled category is a first value, and the label activation values ​​corresponding to the other categories in the K+1 categories besides the labeled category are a second value, and the first value and the second value are different;

[0170] Based on the K+1 labeled activation values ​​corresponding to the predicted known category samples, the K+1 activation values ​​corresponding to the predicted known category samples obtained by the recognizer, and the temperature coefficient, the training loss of the recognizer is calculated; wherein, the temperature coefficient is used to adjust the sharpness of the activation value distribution.

[0171] In some embodiments, the pre-training process of the recognizer is as follows:

[0172] Obtain the pre-training dataset of the recognizer, the pre-training dataset including at least one pre-training sample with a category label, the category label being one of the unknown category and the K known categories;

[0173] The recognizer is trained using the pre-trained dataset to obtain a pre-trained recognizer;

[0174] The pre-trained recognizer is used to identify the predicted known category samples from the open set data.

[0175] This application uses a recognizer to initially identify samples in open-set data to obtain samples with predicted known categories. Then, it performs a secondary screening of these samples based on confidence level to obtain training samples for training the classifier. This application solves the problem that ordinary annotation methods cannot be applied to open-set data, providing a method for identifying and labeling samples with known categories from open-set data. Furthermore, this method utilizes a recognizer for initial identification and then performs a secondary screening based on confidence level, fully ensuring the accuracy of the known category samples selected from the open-set data.

[0176] Please refer to Figure 10 This illustration shows a schematic diagram of a computer device provided in one embodiment of this application. The computer device can be any electronic device with data computing, processing, and storage functions, such as a mobile phone, tablet computer, PC (Personal Computer), or server. This computer device can be implemented as a model training device to implement the open-set data annotation method provided in the above embodiments. Specifically:

[0177] The computer device 1100 includes a central processing unit (such as a CPU, GPU, or FPGA) 1001, a system memory 1004 including RAM (Random-Access Memory) 1002 and ROM (Read-Only Memory) 1003, and a system bus 1005 connecting the system memory 1004 and the central processing unit 1001. The computer device 1100 also includes a basic input / output system 1006 to facilitate information transfer between various devices within the server, and a large-capacity storage device 10010 for storing the operating system 1013, application programs 1014, and other program modules 1015.

[0178] The basic input / output system 1006 includes a display 1008 for displaying information and an input device 1009 for user input, such as a mouse or keyboard. Both the display 1008 and the input device 1009 are connected to the central processing unit 1001 via an input / output controller 1010 connected to the system bus 1005. The basic input / output system 1006 may also include the input / output controller 1010 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1010 also provides output to a display screen, printer, or other types of output devices.

[0179] The mass storage device 10010 is connected to the central processing unit 1001 via a mass storage controller (not shown) connected to the system bus 1005. The mass storage device 1007 and its associated computer-readable media provide non-volatile storage for the computer device 1000. That is, the mass storage device 10010 may include computer-readable media (not shown) such as a hard disk or a CD-ROM (Compact Disc Read-Only Memory) drive.

[0180] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory or other solid-state storage technologies, CD-ROM, DVD (Digital Video Disc) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage medium is not limited to the above-mentioned types. The system memory 1004 and the mass storage device 10010 described above can be collectively referred to as memory.

[0181] According to embodiments of this application, the computer device 1000 can also be connected to a remote computer on a network, such as the Internet. That is, the computer device 1000 can be connected to the network 1012 via the network interface unit 1011 connected to the system bus 1005, or the network interface unit 1011 can be used to connect to other types of networks or remote computer systems (not shown).

[0182] The memory also includes at least one instruction, at least one program, code set, or instruction set, which is stored in the memory and configured to be executed by one or more processors to implement the above-described open set data annotation method.

[0183] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set implements the open set data annotation method provided in the above embodiments when executed by a processor of a computer device.

[0184] Optionally, the computer-readable storage medium may include: ROM (Read-Only Memory), RAM (Random-Access Memory), SSD (Solid State Drives), or optical disc, etc. The random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0185] In an exemplary embodiment, a computer program product or computer program is also provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned open set data annotation method.

[0186] It should be understood that "multiple" as used herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order. In some other embodiments, the steps may not be executed in numerical order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this.

[0187] The above are merely exemplary embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application shall be included within the protection scope of this application.

Claims

1. A method for labeling open set data, characterized in that, The method includes: Obtain an open set of data containing multiple samples, wherein the multiple samples include at least one known category sample and at least one unknown category sample; wherein, the known category sample refers to the sample whose true category belongs to K known categories, and the unknown category sample refers to the sample whose true category does not belong to the K known categories, where K is a positive integer, and the samples in the open set of data are image samples; For the target sample in the open set data, the target sample is input into the recognizer; the recognizer obtains K+1 activation values ​​corresponding to the target sample, and the K+1 activation values ​​correspond one-to-one with K+1 categories, the K+1 categories including unknown categories and the K known categories; If the K+1 activation values ​​meet the conditions, the target sample is determined to be the known category sample; from the open set data, N samples determined to be the known category samples are selected to obtain N predicted known category samples, where the predicted known category samples refer to the samples that the recognizer identifies as the known category samples, and N is a positive integer; The distribution of the maximum activation values ​​corresponding to the N predicted known category samples is modeled using a Gaussian mixture model (GMM) to obtain the probability values ​​corresponding to the N predicted known category samples. The probability values ​​are used to characterize the confidence that the predicted known category sample belongs to the known category. From the N predicted known category samples, the predicted known category samples that satisfy the target condition based on the probability value are used to obtain the training samples of the classifier, which is used to classify the K known categories.

2. The method according to claim 1, characterized in that, If the K+1 activation values ​​meet the conditions, then the target sample is determined to be a known category sample, including: Determine the maximum activation value among the K+1 activation values; If the category corresponding to the maximum activation value belongs to the K known categories, then the target sample is determined to be a sample of the known category.

3. The method according to claim 1, characterized in that, The Gaussian Mixture Model (GMM) is used to model the distribution of the maximum activation values ​​corresponding to the N predicted known class samples, obtaining the probability values ​​corresponding to the N predicted known class samples, including: For the target known category among the K known categories, select M predicted known category samples belonging to the target known category from the N predicted known category samples, where M is a positive integer less than or equal to N; For each of the M predicted known category samples, the activation value corresponding to the target known category is obtained from the K+1 activation values ​​corresponding to the predicted known category sample, thus obtaining M activation values; The GMM is used to model the M activation values ​​to obtain the probability values ​​corresponding to the M predicted known category samples respectively; wherein, the probability value corresponding to the i-th predicted known category sample among the M predicted known category samples is used to characterize the confidence that the i-th predicted known category sample belongs to the target known category, and i is a positive integer less than or equal to M.

4. The method according to claim 1, characterized in that, The step of obtaining training samples for the classifier from the N predicted known category samples based on the probability values ​​satisfying the target condition includes: Obtain the labeled category corresponding to the predicted known category sample whose probability value satisfies the target condition, wherein the labeled category is one of the K+1 categories; The predicted known category samples whose labeled categories belong to the K known categories are used as training samples for the classifier.

5. The method according to claim 4, characterized in that, After obtaining the labeled category corresponding to the predicted known category sample whose probability value meets the condition, the method further includes: Based on the labeled category corresponding to the predicted known category sample, and the K+1 activation values ​​corresponding to the predicted known category sample obtained by the recognizer, the training loss of the recognizer is calculated; The parameters of the recognizer are adjusted based on the training loss of the recognizer to obtain an updated recognizer; wherein the updated recognizer is used to identify new samples of the predicted known class from the open set data.

6. The method according to claim 5, characterized in that, The step of calculating the training loss of the recognizer based on the labeled category corresponding to the predicted known category sample and the K+1 activation values ​​corresponding to the predicted known category sample obtained by the recognizer includes: Based on the labeled categories corresponding to the predicted known category samples, K+1 label activation values ​​corresponding to the predicted known category samples are determined; wherein, the label activation value corresponding to the labeled category is a first value, and the label activation values ​​corresponding to the other categories in the K+1 categories besides the labeled category are a second value, and the first value and the second value are different; Based on the K+1 labeled activation values ​​corresponding to the predicted known category samples, the K+1 activation values ​​corresponding to the predicted known category samples obtained by the recognizer, and the temperature coefficient, the training loss of the recognizer is calculated; wherein, the temperature coefficient is used to adjust the sharpness of the activation value distribution.

7. The method according to any one of claims 1 to 6, characterized in that, The pre-training process of the recognizer is as follows: Obtain the pre-training dataset of the recognizer, the pre-training dataset including at least one pre-training sample with a category label, the category label being one of the unknown category and the K known categories; The recognizer is trained using the pre-trained dataset to obtain a pre-trained recognizer; The pre-trained recognizer is used to identify the predicted known category samples from the open set data.

8. A labeling device for open set data, characterized in that, The device includes: The sample acquisition module is used to acquire open set data containing multiple samples, wherein the multiple samples include at least one known category sample and at least one unknown category sample; wherein, the known category sample refers to the sample whose true category belongs to K known categories, and the unknown category sample refers to the sample whose true category does not belong to the K known categories, where K is a positive integer, and the samples in the open set data are image samples; The sample recognition module is used to input the target sample in the open set data into the recognizer; and obtain K+1 activation values ​​corresponding to the target sample through the recognizer. The K+1 activation values ​​correspond one-to-one with K+1 categories, and the K+1 categories include unknown categories and the K known categories. The sample recognition module is further configured to determine the target sample as the known category sample if the K+1 activation values ​​meet the conditions; and to select N samples determined as the known category samples from the open set data to obtain N predicted known category samples, wherein the predicted known category samples refer to the samples identified by the recognizer as the known category samples, and N is a positive integer; The sample selection module is used to model the distribution of the maximum activation values ​​corresponding to the N predicted known category samples using GMM, and obtain the probability values ​​corresponding to the N predicted known category samples respectively. The probability values ​​are used to characterize the confidence that the predicted known category sample belongs to the known category. The sample selection module is further configured to obtain training samples for the classifier from the N predicted known category samples based on the probability values ​​that satisfy the target conditions, and the classifier is used to classify the K known categories.

9. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one program, the at least one program being loaded and executed by the processor to implement the method as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The storage medium stores at least one program segment, which is loaded and executed by a processor to implement the method as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium, which a processor reads from and executes to implement the method as described in any one of claims 1 to 7.

Images