An open set sample decomposition method, device, and related equipment

Abstract

Embodiments of this application disclose an open-set sample decomposition method, apparatus, and related devices, which specifically relate to the field of artificial intelligence technology and can be applied to the field of maps. The method includes: obtaining open-set sample data and closed-set sample data; extracting open-set features corresponding to the open-set sample data and extracting closed-set features corresponding to the closed-set sample data, and combining the open-set features and the closed-set features into target features; furthermore, inputting the open-set features into a multi-granularity decomposer to obtain multi-granularity subclasses; moreover, performing matching processing on the multi-granularity subclasses to obtain multi-granularity pseudo-labels; and determining the open-set structure information corresponding to the multi-granularity subclasses; through a sequence structure learner, training the target features and the multi-granularity pseudo-labels to obtain a decomposition result; finally, updating the open-set structure information according to the decomposition result. Using this application is beneficial to improving the recognition ability for open-set samples.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular, to an open-set sample decomposition method, apparatus, and related devices. Background Art

[0002] Currently, there are two goals for open-set recognition tasks: accurately distinguishing closed-set sample categories and accurately identifying open-set samples that are not included in the closed-set categories. Commonly used methods often do not use open-set samples. Even if open-set samples are used, all open-set samples are simply regarded as a single "other" class for training. However, in actual scenarios, open-set samples are easy to collect and ubiquitous. Directly ignoring the existence of open-set samples not only increases the difficulty of open-set recognition tasks but also leads to data waste, which is not conducive to the training of today's data-driven neural network models. Summary of the Invention

[0003] Embodiments of this application provide an open-set sample decomposition method, apparatus, and related devices, which are beneficial to improving the utilization rate of data and are also beneficial to improving the recognition ability for open-set samples.

[0004] One aspect of this application provides an open-set sample decomposition method, including:

[0005] Obtain open-set sample data and closed-set sample data; extract open-set features corresponding to the open-set sample data, and extract closed-set features corresponding to the closed-set sample data, and combine the open-set features and the closed-set features into target features;

[0006] Input the open-set features into a multi-granularity decomposer to obtain multi-granularity subclasses; perform matching processing on the multi-granularity subclasses to obtain multi-granularity pseudo-labels; and determine open-set structure information corresponding to the multi-granularity subclasses;

[0007] Train the target features and the multi-granularity pseudo-labels through a sequence structure learner to obtain a decomposition result;

[0008] Update the open-set structure information according to the decomposition result.

[0009] Among them, the step of inputting the open-set features into a multi-granularity decomposer to obtain multi-granularity subclasses specifically includes the following steps:

[0010] Cluster the open-set features with a preset step size through the multi-granularity decomposer to obtain a plurality of clusters, and the plurality of clusters form the multi-granularity subclasses;

[0011] The step of clustering the open-set features with a preset step size to obtain a plurality of clusters includes:

[0012] If the open set features are clustered twice with the preset step size, a first type of cluster result and the second type of cluster result are obtained. The first type of cluster result includes multiple first type of clusters, and the second type of cluster result includes multiple second type of clusters. The multiple first type of clusters and the multiple second type of clusters form the multiple clusters.

[0013] Among them, the matching process for the multi-granularity subclasses to obtain multi-granularity pseudo-labels specifically includes the following steps:

[0014] The multiple first type of clusters are matched one by one with the multiple second type of clusters to obtain multiple pairs of matching clusters. Among them, the matching clusters are composed of a first type of cluster and a second type of cluster;

[0015] Pseudo-labels are assigned to the sample data corresponding to the sample features in the multiple pairs of matching clusters to obtain the multi-granularity pseudo-labels.

[0016] Among them, the one-by-one matching of the multiple first type of clusters with the multiple second type of clusters specifically includes the following steps:

[0017] If there are common samples between the first type of cluster i in the multiple first type of clusters and the second type of cluster j in the multiple second type of clusters, and compared with other second type of clusters except the second type of cluster j, the second type of cluster j has the most common samples with the first type of cluster i, then it is determined that the first type of cluster i and the second type of cluster j are a pair of matching clusters, where the first type of cluster i is the i-th first type of cluster in the multiple clusters, and the second type of cluster j is the j-th second type of cluster in the multiple second type of clusters;

[0018] After determining that the first type of cluster i and the second type of cluster j are a pair of matching clusters, continue to match the first type of cluster i + 1 with other second type of clusters except the second type of cluster j to obtain multiple pairs of matching clusters.

[0019] Among them, the matching process for the multi-granularity subclasses to obtain multi-granularity pseudo-labels specifically includes the following steps:

[0020] The target feature is embedded into multiple label spaces of the sequence structure learner through a multi-label embedder;

[0021] The multi-granularity pseudo-labels are trained through the multiple label spaces to obtain a decomposition result composed of the multi-granularity pseudo-labels and multiple trained multi-granularity pseudo-labels.

[0022] Among them, the embedding of the target feature into multiple label spaces of the sequence structure learner through a multi-label embedder specifically includes the following steps:

[0023] Determine the quantity corresponding to the multi-granularity pseudo-labels;

[0024] Determine the number of fully connected layers according to the quantity, and obtain the fully connected layers of the multi-layer structure, where the number of fully connected layers corresponds to the quantity of the label space;

[0025] Embed the target feature into multiple label spaces of the sequence structure learner through the fully connected layers of the multi-layer structure and a preset encoder.

[0026] Specifically, it further includes the following steps:

[0027] Determine the true label corresponding to the open-set sample data;

[0028] Obtain a preset cross-entropy loss function, and train the classification task corresponding to the true label and the multi-label classification task corresponding to the multi-granularity pseudo label according to the preset cross-entropy loss function, to obtain the prediction result corresponding to the true label and the prediction result corresponding to the multi-granularity label;

[0029] Optimize the preset cross-entropy loss function according to the prediction result corresponding to the true label and the prediction result corresponding to the multi-granularity label.

[0030] On the other hand, the present application provides an open-set sample decomposition device, including:

[0031] A first acquisition module, configured to acquire open-set sample data and closed-set sample data; extract the open-set feature corresponding to the open-set sample data, and extract the closed-set feature corresponding to the closed-set sample data, and combine the open-set feature and the closed-set feature into a target feature;

[0032] A matching module, configured to input the open-set feature into a multi-granularity decomposer to obtain multi-granularity subclasses; perform matching processing on the multi-granularity subclasses to obtain multi-granularity pseudo labels; and determine the open-set structure information corresponding to the multi-granularity subclasses;

[0033] A training module, configured to train the target feature and the multi-granularity pseudo label through a sequence structure learner to obtain a decomposition result;

[0034] An updating module, configured to update the open-set structure information according to the decomposition result.

[0035] Wherein, the matching module includes:

[0036] A clustering unit, configured to cluster the open-set feature through the multi-granularity decomposer at a preset step length to obtain multiple clusters, and the multiple clusters form the multi-granularity subclasses;

[0037] Among them, the clustering unit is used to perform two clusterings on the open-set features with the preset step size to obtain a first cluster result and a second cluster result. The first cluster result includes multiple first clusters, and the second cluster result includes multiple second clusters. The multiple first clusters and the multiple second clusters form the multiple clusters.

[0038] Among them, the matching module may further include:

[0039] A matching unit for one-to-one matching of the multiple first clusters and the multiple second clusters to obtain multiple pairs of matching clusters, where the matching clusters are composed of a first cluster and a second cluster;

[0040] A labeling unit for labeling the sample data corresponding to the sample features in the multiple pairs of matching clusters with pseudo-labels to obtain the multi-granularity pseudo-labels.

[0041] Among them, the matching unit includes:

[0042] A first determination subunit for determining that the first cluster i in the multiple first clusters and the second cluster j in the multiple second clusters are a pair of matching clusters if there are common samples between the first cluster i and the second cluster j, and the second cluster j has the most common samples with the first cluster i compared to other second clusters except the second cluster j. Here, the first cluster i is the i-th first cluster in the multiple clusters, and the second cluster j is the j-th second cluster in the multiple second clusters;

[0043] A matching subunit for, after determining that the first cluster i and the second cluster j are a pair of matching clusters, continuing to match the first cluster i + 1 with other second clusters except the second cluster j to obtain multiple pairs of matching clusters.

[0044] Among them, the training module includes:

[0045] An embedding unit for embedding the target feature into multiple label spaces of the sequence structure learner through a multi-label embedder;

[0046] A training unit for training the multi-granularity pseudo-labels through the multiple label spaces to obtain a decomposition result composed of the multi-granularity pseudo-labels and multiple trained multi-granularity pseudo-labels.

[0047] Among them, the embedding unit includes:

[0048] A second determination subunit for determining the quantity corresponding to the multi-granularity pseudo-labels;

[0049] A third determination subunit, configured to determine the number of fully connected layers according to the quantity, so as to obtain fully connected layers of a multi-layer structure, where the number of fully connected layers corresponds to the quantity of the label spaces;

[0050] An embedding subunit, configured to embed the target features into multiple label spaces of the sequence structure learner respectively through the fully connected layers of the multi-layer structure and a preset encoder.

[0051] Wherein, the apparatus further includes: a determination module, a second acquisition module, and an optimization module;

[0052] The determination module is configured to determine the true label corresponding to the open-set sample data;

[0053] The second acquisition module is configured to acquire a preset cross-entropy loss function, and train a classification task corresponding to the true label and a multi-label classification task corresponding to the multi-granularity pseudo-labels according to the preset cross-entropy loss function, so as to obtain a prediction result corresponding to the true label and a prediction result corresponding to the multi-granularity labels;

[0054] The optimization module is configured to optimize the preset cross-entropy loss function according to the prediction result corresponding to the true label and the prediction result corresponding to the multi-granularity labels.

[0055] Another aspect of the present application provides a terminal device, including: a processor and a memory;

[0056] The processor is connected to the memory, wherein the memory is used to store program codes, and the processor is used to call the program codes to execute the method in one aspect of the embodiments of the present application.

[0057] Another aspect of the embodiments of the present application provides a computer storage medium, where the computer storage medium stores a computer program, and the computer program includes program instructions, and when the program instructions are executed by a processor, the method in one aspect of the embodiments of the present application is executed.

[0058] Another aspect of the embodiments of the present application provides a computer program product or a computer program, the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided in the above-mentioned one aspect or various optional manners of one aspect.

[0059] In the embodiments of the present application, open-set sample data and closed-set sample data are obtained; open-set features corresponding to the open-set sample data are extracted, and closed-set features corresponding to the closed-set sample data are extracted, and the open-set features and the closed-set features are combined into target features; furthermore, the open-set features are input into a multi-granularity decomposer to obtain multi-granularity subclasses; moreover, matching processing is performed on the multi-granularity subclasses to obtain multi-granularity pseudo-labels; and open-set structure information corresponding to the multi-granularity subclasses is determined; through a sequence structure learner, the target features and the multi-granularity pseudo-labels are trained to obtain a decomposition result; finally, the open-set structure information is updated according to the decomposition result. In this way, the open-set sample data can be decomposed by the multi-granularity decomposer; and through the sequence structure learner, combined with the closed-set samples, the multi-granularity pseudo-labels obtained by the multi-granularity decomposer are learned to obtain a decomposition result, that is, the open-set structure information obtained in the current round of training, and the open-set structure information corresponding to the above open-set sample data is updated according to the trained open-set structure information; by repeating this process, the open-set samples can be dynamically decomposed according to the learning progress of the closed-set samples, and the visual knowledge in the closed-set information can be transferred to the open-set samples. In addition, the multi-granularity decomposer can also be used to decompose the open-set samples to obtain multi-granularity subclasses, realizing the subdivision of the open-set samples, which is beneficial to improving the recognition ability of the open-set samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0061] Figure 1a is a schematic structural diagram of an open-set sample decomposition system provided by an embodiment of the present application;

[0062] Figure 1b is a schematic diagram of open-set sample data provided by an embodiment of the present application;

[0063] Figure 1c is a schematic diagram of closed-set sample data provided by an embodiment of the present application;

[0064] Figure 1d is a schematic diagram of a multi-granularity decomposition scenario provided by an embodiment of the present application;

[0065] Figure 1e is a schematic structural diagram of a sequence structure learner provided by an embodiment of the present application;

[0066] Figure 2 is a schematic flowchart of an open-set sample decomposition method provided by an embodiment of the present application;

[0067] Figure 3a is a schematic flowchart of another open-set sample decomposition method provided by an embodiment of the present application;

[0068] Figure 3b is a schematic diagram of the scenario of an open-set sample decomposition method provided by an embodiment of the present application;

[0069] Figure 4 is a schematic structural diagram of an open-set sample decomposition device provided by an embodiment of the present application;

[0070] Figure 5 is a schematic structural diagram of a terminal device provided by an embodiment of the present application. Detailed implementation manners

[0071] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0072] Please refer to Figure 1a , which is a schematic structural diagram of an open-set sample decomposition system provided by an embodiment of the present application. As Figure 1aAs shown, the structure 100 may at least include three models such as a multi-granularity decomposer 101, a sequence structure learner 102, and a convolutional neural network 103. The open-set sample decomposition method provided by the embodiments of the present application can be applied to the above open-set sample decomposition system, which relates to the field of Artificial Intelligence (AI) technology. AI is a theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a way similar to human intelligence. Artificial intelligence also studies the design principles and implementation methods of various intelligent machines to enable the machines to have the functions of perception, reasoning, and decision-making. Artificial intelligence technology is an interdisciplinary subject that involves a wide range of fields, including both hardware-level technologies and software-level technologies. The basic technologies of artificial intelligence generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. The software technologies of artificial intelligence mainly include several major directions such as computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning. Among them, the embodiments of the present application specifically relate to fields such as image processing, image recognition, image semantic understanding, image retrieval, Optical Character Recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, autonomous driving, and intelligent transportation in computer vision technology corresponding to artificial intelligence technology; specifically, it can involve the recognition of open-set sample data in the training set. The goals of the open-set recognition task are two. One is to accurately distinguish the closed-set sample categories, and the other is to accurately identify the open-set samples that are not included in the closed-set classes for tagging the sample data to classify and identify the sample data corresponding to different tags, thereby obtaining new training sample data, which is beneficial to subsequent image recognition.

[0073] Among them, the above-mentioned closed-set sample data may mean that each category in it has a specific label, does not include unknown categories, and the categories in the closed-set sample data are consistent. That is to say, the categories of the images in all the closed-set sample data are known, and there are no images of unknown categories. The above-mentioned open-set sample data may mean that it contains categories not in the test set; for example: when using the test set for testing, an image that does not belong to the known categories in the sample data can be input; when the open-set sample data includes multiple types of data, it can reflect data diversity. As Figure 1b shown, it is a schematic diagram of a kind of open-set sample data, such as Figure 1cAs shown, it is a schematic diagram of closed-set sample data. As shown in the figure, the above-mentioned closed-set sample data may include images of cats, dogs, etc. Different from the closed-set sample data, the open-set sample data may further include images of rabbits, horses, etc. in addition to the images of cats, dogs, etc. It can be seen that the open-set sample data is more diverse than the closed-set sample data, and the open-set sample data is easy to collect and ubiquitous. Ignoring the existence of the open-set sample directly not only increases the difficulty of the open-set recognition task but also leads to a waste of data. In the embodiments of the present application, through multiple (in the embodiments of the present application, multiple may refer to two or more, and will not be elaborated hereinafter) modules in the above-mentioned open-set sample system, the above-mentioned open-set sample data can be used to realize the recognition or distinction of open-set samples and closed-set samples, and the utilization rate of data can be improved.

[0074] In specific implementation, open-set sample data and closed-set sample data can be obtained; the open-set features corresponding to the open-set sample data are extracted through the convolutional neural network 103, and the closed-set features corresponding to the closed-set sample data are extracted, and the open-set features and the closed-set features are combined into target features; furthermore, the open-set features are input into the multi-granularity decomposer 101 to obtain a set of multi-granularity subclasses; moreover, the multi-granularity subclasses are subjected to matching processing to obtain multi-granularity pseudo-labels; and the open-set structure information corresponding to the multi-granularity subclasses is determined; through the sequence structure learner 102, the target features and the multi-granularity pseudo-labels are trained to obtain a decomposition result; finally, the open-set structure information is updated according to the decomposition result. In this way, the updated open-set structure information can be continuously input into the above-mentioned multi-granularity decomposer 101, and the above operation steps are repeated, and the open-set sample data can be dynamically decomposed according to the learning progress of the closed-set sample data in the sequence structure learner 102. At the same time, the visual knowledge in the closed-set information can be migrated to the open-set sample data. In addition, the multi-granularity decomposer 102 can also be used to decompose the open-set sample to obtain one or more multi-granularity subclasses, realizing the subdivision of the open-set sample data, which is beneficial to improving the recognition ability of the open-set sample.

[0075] Among them, as Figure 1dAs shown in the figure, it is a schematic diagram of a multi-granularity decomposition scenario. Through clustering, the corresponding features can be divided into multiple granularities, such as coarse granularity, intermediate granularity, and fine granularity, etc. Among them, taking a puppy picture as an open-set sample data as an example, the features corresponding to the open-set sample data can be input into the multi-granularity decomposer 101. Through clustering, its features are divided into 3 granularities. Among them, under the coarse-granularity division, it can include: head features, body features, and tail features; for the head features, intermediate-granularity division can be carried out, which can include: ear features, eye features, nose features, and mouth features; for the eye features, fine-granularity division can be carried out, which can include: pupil features, eye contour features, etc. In this way, according to the granularity from large to small, the open-set sample data can be successively divided into subclasses of coarse granularity, subclasses of intermediate granularity, and subclasses of fine granularity. The granularities corresponding to different subclasses are different. In this way, the open-set features corresponding to the open-set sample can be decomposed to obtain multi-granularity subclasses.

[0076] Among them, as Figure 1e shown, it is a schematic diagram of the structure of a sequence structure learner. As shown in the figure, the sequence structure learner 102 can include a preset encoder, multiple label spaces, and a fully connected layer not shown in the figure. Among them, each small square can represent a label space, and the fully connected layer is connected to the preset encoder and the label space.

[0077] In the embodiment of the present application, the above sequence structure learner 102 is built on a general self-training framework and can be trained in multiple rounds by using its past state experiences (open-set structure information). The above preset encoder is a multi-layer structure, and each layer can be connected to multiple label spaces. In order to use the open-set structure information obtained in the (t - 1)-th round to assist the learning of the current open-set structure information, a Transformer encoder can be used. Transformer is a model that accelerates deep learning algorithms based on the attention mechanism. The Transformer encoder consists of a group of encoders and a group of decoders. Self-attention can exchange information between the previous time step and the current time step in a parallel manner. In this way, the interaction between the closed-set space and the open-set space can be realized, that is to say, the self-attention mechanism realizes global information interaction among all label spaces. Through the sequence structure learner 102, the multi-granularity pseudo-labels after training, that is, the decomposition results, can be obtained. The information between each pseudo-label after training can be interacted, that is to say, information interaction can be realized in the label space. For example, the first pseudo-label after training until the last pseudo-label after training will all affect each other.

[0078] For example, the embodiments of the present application can be applied to content review services. Content review is a detection technology based on images, texts, videos, etc., which can automatically detect relevant content such as advertisements to achieve content review of pictures, texts, and videos uploaded by users. Among them, in this content review service, normal samples are often regarded as an entire class for training. The intra-class differences of normal samples are huge and difficult to be grasped by the model. Therefore, normal samples (for example, non-advertisement and other relevant content) can form an open-set sample, and the open-set sample decomposition method described in the embodiments of the present application can be used to decompose the above normal samples to obtain additional information (for example, in the social application scenario, it can provide richer materials for data classification, including WeChat business advertisements, various contact information content), so as to assist the training of content review models based on classification, etc. At the same time, it is also beneficial to improve the recognition ability of the content review model based on classification for open-set samples to improve the accuracy of content review.

[0079] For another example, the embodiments of the present application can also be applied to the field of maps. For example, open-set sample data and closed-set sample data can be preset. The open-set sample data can include pictures of various buildings, and the closed-set sample data can include pictures of known buildings. The visual knowledge in the closed-set information can be transferred to the open-set sample data through the open-set sample decomposition method provided by the embodiments of the present application. In addition, a multi-granularity decomposer can be used to decompose the open-set samples to obtain one or more multi-granularity subclasses, realizing the subdivision of the open-set sample data, so as to assign multi-granularity pseudo-labels to it, which is beneficial to improving the recognition ability of open-set samples.

[0080] Among them, the solution provided by the embodiments of the present application involves technologies such as open-set recognition in computer vision, which will be specifically described through the following embodiments:

[0081] Please refer to Figure 2 , which is a schematic flowchart of an open-set sample decomposition method provided by the embodiments of the present application. As Figure 2 shown, the open-set sample decomposition method may include:

[0082] Step S101, obtain open-set sample data and closed-set sample data; extract the open-set features corresponding to the open-set sample data, and extract the closed-set features corresponding to the closed-set sample data, and combine the open-set features and the closed-set features into target features.

[0083] Specifically, the open-set samples may include open-set sample data, and the closed-set samples may include closed-set sample data; the electronic device can obtain open-set sample data and closed-set sample data (the open-set sample data and the closed-set sample data are as described above Figure 1b and 1c(as described in the corresponding embodiments), and extract the open-set features corresponding to the open-set sample data and the closed-set features corresponding to the closed-set sample data through a preset network (e.g., a convolutional neural network), and combine the open-set features and the closed-set features into target features.

[0084] Step S102: Input the open-set features into a multi-granularity decomposer to obtain multi-granularity subclasses; perform matching processing on the multi-granularity subclasses to obtain multi-granularity pseudo-labels; and determine the open-set structure information corresponding to the multi-granularity subclasses.

[0085] Among them, due to the diversity of the open-set sample data, the open-set sample data can be divided into multiple non-overlapping subsets through a multi-granularity decomposer, that is, multiple multi-granularity subclasses are obtained. Each multi-granularity subclass may include at least one of the following: sample data of multiple granularities such as fine-grained, medium-grained, and coarse-grained. For example, if the open-set sample data includes pictures of cats, dogs, horses, etc., the features corresponding to parts such as ears, eyes, and noses can be used as medium-grained, the features corresponding to parts such as heads, upper limbs, and lower limbs can be used as coarse-grained, and the features of parts such as eyelashes, eye opening degree, pupils, ear contours, ear sizes, nose tips, nasal bridges, and nostrils can be used as fine-grained.

[0086] Specifically, the multi-granularity decomposer can be used to cluster the open-set features with a preset step size to obtain multiple clusters, and the multiple clusters form multi-granularity subclasses; the above clustering method may include K-means clustering or a representative density-based clustering algorithm (Density-Based Spatial Clustering of Applications with Noise, DBSCAN) clustering method, etc. In this way, the open-set features can be divided into multi-granularity subclasses, and each subclass may include feature data of multiple granularities. For example, for a certain subclass, it may include features of multiple granularities such as ears, ear contours, and ear sizes.

[0087] When using the K-means clustering method, although K-means is an extensible clustering algorithm suitable for large-scale data sets, we still need to select the number of clusters N in advance. And to prevent over-clustering and under-clustering problems from a single N, an appropriate interval N min ,N max before can be set as the preset step size, so as to determine the number of clusters N to perform multiple clusterings on the above open-set sample data, thereby dividing the open-set samples into multi-granularity subclasses. step

[0088] ​Furthermore, clustering the open set features with a preset step size to obtain multiple clusters may specifically include the following steps: In this application, only two clusters are taken as an example. If the open set features are clustered twice with a preset step size, a first cluster result and a second cluster result can be obtained. The first cluster result includes multiple first clusters, and the second cluster result includes multiple second clusters. The multiple first clusters and the multiple second clusters form multiple clusters.

[0089] For example, given the open set sample W = {w1,…,w n}, first define the consistency of two K-means clustering results. Let N be the number of K-means clusters, and assume the clustering results: The first cluster result: The second cluster result: where S is the set of samples in a cluster in the clustering result. Furthermore, a matching function g can be obtained through the Hungarian algorithm to match the two K-means clustering results, where and are a pair of matching clusters, then the consistency of the clustering results R1 and R2 is defined as:

[0090]

[0091] Due to the randomness of K-means clustering, directly using the single clustering result as a pseudo-label will bring large variance. Therefore, through bipartite graph matching, only the sample data that match the two clustering results are labeled with multi-granularity pseudo-labels. Specifically, the multiple first clusters are matched one by one with the multiple second clusters to obtain multiple pairs of matching clusters, where the matching clusters are composed of a first cluster and a second cluster; the sample data corresponding to the sample features in the multiple pairs of matching clusters are labeled with the same pseudo-label. In this way, multi-granularity pseudo-labels can be obtained after matching the multiple first clusters and the second clusters. A subcategory can correspond to a set of attributes. Different parts can be used as different subcategories. For example, parts such as the head, upper limbs, and lower limbs are used as different subcategories. A cluster can include sample data of multiple granularities. It can be determined that the first cluster includes features corresponding to sample data such as ears, ear contours, and ear sizes, and the second cluster includes features corresponding to sample data such as upper limbs, ears, ear contours, ear sizes, and eyes. It can be further determined that there are common sample data between the sample data corresponding to the two clusters, such as sample data such as ears, ear contours, and ear sizes. Then, it can be determined that the above first cluster and the second cluster form a matching pair, and the open set sample data corresponding to the first cluster and the second cluster are labeled with an approximate label, that is, a multi-granularity pseudo-label. For example, the first cluster and the second cluster can both be labeled with the multi-granularity pseudo-label of cat. Furthermore, after dividing the above open set samples into multi-granularity self by a multi-granularity decomposer, the open set structure information corresponding to the above clusters can be determined.

[0092] Among them, the open set structure information can be composed of the above-mentioned multi-granularity pseudo-labels, and the multi-granularity pseudo-labels can reflect the pseudo-labels under the characteristics of multiple granularities.

[0093] In addition, the multiple first-type clusters and the multiple second-type clusters can be matched one by one in a bipartite graph manner. Specifically, the multiple first-type clusters can be respectively matched with the multiple second-type clusters one by one. It can start from the first first-type cluster among the multiple first-type clusters and select the second-type cluster that matches it from the multiple second-type clusters. Specifically, if there are common samples between the first-type cluster i among the multiple first-type clusters and the second-type cluster j among the multiple second-type clusters, and compared with other second-type clusters except the second-type cluster j, the second-type cluster j has the most common samples with the first-type cluster i, then it is determined that the first-type cluster i and the second-type cluster j are a pair of matching clusters, where the first-type cluster i is the i-th first-type cluster among the multiple clusters, and the second-type cluster j is the j-th second-type cluster among the multiple second-type clusters; after determining that the first-type cluster i and the second-type cluster j are a pair of matching clusters, the above-mentioned first-type cluster i and second-type cluster j can be removed from the subsequent matching process, and continue to match the first-type cluster i + 1 with the remaining second-type clusters except the second-type cluster j. In this way, multiple pairs of matching clusters can be obtained through repeated cycles.

[0094] Among them, both the above-mentioned i and j are positive integers.

[0095] Step S103, through a sequence structure learner, train the target feature and the multi-granularity pseudo-labels to obtain a decomposition result.

[0096] Specifically, a multi-label embedder can be used to embed the target feature into multiple label spaces of the sequence structure learner; train the multi-granularity pseudo-labels through the multiple label spaces to obtain a decomposition result composed of the multi-granularity pseudo-labels and multiple trained multi-granularity pseudo-labels. This decomposition result is actually a sequence, and this sequence can be obtained by combining the multi-granularity pseudo-labels obtained through a multi-granularity decomposer and the multiple trained multi-granularity pseudo-labels obtained by the sequence structure learner.

[0097] Among them, the sequence structure learner is built on a general self-training framework, where the model experiences multiple training stages by using its past states. As the training progresses, the decomposition results of different training stages can form a sequence, which can be used as the sequence structure of the open set samples. Furthermore, a set (multiple) of multi-granularity pseudo-labels obtained by decomposing the sequence structure of the open set samples through a multi-granularity decomposer can be put into the sequence structure learner for training. After t rounds of training, a decomposition result is obtained, that is, new open set structure information, that is, a sequence structure composed of multiple groups of multi-granularity pseudo-labels after training and a group of multi-granularity pseudo-labels input into the sequence structure learner. This process repeats in a cycle.

[0098] Further, embedding the target features into multiple label spaces of the sequence structure learner through a multi-label embedder may include the following steps: determining the number of corresponding multi-granularity pseudo-labels; according to the number, determining the number of layers of the fully connected layer to obtain a fully connected layer with a multi-layer structure, where the number of layers of the fully connected layer corresponds to the number of label spaces; through the fully connected layer with a multi-layer structure and a preset encoder, respectively embedding the target features into multiple label spaces of the sequence structure learner.

[0099] Specifically, when performing the t-th round of training through the sequence structure learner, the open-set samples correspond to 1 + t × N g labels, which may specifically be composed of 1 true label (the true label refers to the original task learning or the learned label, for example, the label of the problem to be solved (such as whether it is an open-set sample, etc.)) and t × N g multi-granularity pseudo-labels. N g is the number of multi-granularity labels (labels corresponding to different granularities) in each stage, so as to determine the number of corresponding multi-granularity pseudo-labels, and then different fully connected layers can be used to embed the target features (including open-set features and closed-set features) into different label spaces:

[0100] h i = fc i (x), i = 0, 1, …, t × N g ;

[0101] where x refers to any sample; fc i is a fully connected layer for h i , and further, a fully connected layer with a multi-layer structure can be obtained.

[0102] Among them, the sequence structure of the above open-set samples is affected by the fully connected layer of the multi-layer architecture. Denote the sequence structure of the open-set samples as In order to utilize the structural information of the previously learned sequence structure (i.e., the structural information corresponding to the sequence structure during the (t - 1)-th round of training) to assist the current learning, the target features can be further embedded into multiple label spaces (such as the label spaces in ) in the sequence structure learner through a preset encoder (such as a Transformer encoder). The self-attention of this preset encoder can exchange information between the previous time step and the current time step in a parallel manner: Figure 1e

[0103] H (L) = Transformer-Encoder L (H (L-1) );

[0104] where L is the number of layers of the Transformer. Further, a series of H(L) Perform parallel processing to embed the target features into multiple label spaces of the sequence structure learner respectively. In this way, the prediction of the sequence structure corresponding to the current open-set sample can refer to the previously trained sequence structure as a warm start. The prediction of the sequence structure corresponding to the current open-set sample can benefit from the information of the label classification at the beginning, and the process of the label classification at the beginning can also benefit from the sequence structure corresponding to the current open-set sample. That is to say, global information interaction between the open-set space and the closed-set space can be realized among the label spaces.

[0105] In addition, the above-mentioned preset encoder can also attempt to model the sequence structure using a Long Short-Term Memory (LSTM) network according to the data scale of the open-set samples. For small-scale data sets, the modeling efficiency of LSTM may be higher, which can effectively avoid overfitting.

[0106] In the embodiment of the present application, open-set sample data and closed-set sample data are obtained; open-set features corresponding to the open-set sample data are extracted, and closed-set features corresponding to the closed-set sample data are extracted, and the open-set features and the closed-set features are combined into target features; furthermore, the open-set features are input into a multi-granularity decomposer to obtain multi-granularity subclasses; moreover, matching processing is performed on the multi-granularity subclasses to obtain multi-granularity pseudo-labels; and open-set structure information corresponding to the multi-granularity subclasses is determined; through a sequence structure learner, the target features and the multi-granularity pseudo-labels are trained to obtain a decomposition result; finally, the open-set structure information is updated according to the decomposition result. The open-set samples can be decomposed by the multi-granularity decomposer; and through the sequence structure learner, combined with the closed-set samples, the multi-granularity pseudo-labels obtained by the multi-granularity decomposer are learned to obtain a decomposition result, that is, the open-set structure information obtained by the current multi-round training, and the open-set structure information corresponding to the above open-set samples is updated according to the trained open-set structure information; in this way, by cycling, the open-set samples can be dynamically decomposed according to the learning progress of the closed-set samples, and the visual knowledge in the closed-set information can be transferred to the open-set samples. In addition, the multi-granularity decomposer can also be used to decompose the open-set samples to obtain multi-granularity subclasses, realizing the subdivision of the open-set samples, which is beneficial to improving the recognition ability of the open-set samples.

[0107] Further, please refer to Figure 3a , which is a schematic flowchart of another open-set sample decomposition method provided by the embodiment of the present application. As Figure 3a shown, the above open-set sample decomposition method may include:

[0108] Step S201, obtain open-set sample data and closed-set sample data; extract open-set features corresponding to the open-set sample data, and extract closed-set features corresponding to the closed-set sample data, and combine the open-set features and the closed-set features into target features;

[0109] Step S202: Input the open-set features into a multi-granularity decomposer to obtain multi-granularity subclasses; perform matching processing on the multi-granularity subclasses to obtain multi-granularity pseudo-labels; and determine the open-set structure information corresponding to the multi-granularity subclasses.

[0110] Step S203: Through a sequence structure learner, train the target features and the multi-granularity pseudo-labels to obtain a decomposition result.

[0111] Step S204: Update the open-set structure information according to the decomposition result.

[0112] Among them, the specific implementation manners of Step S201 - Step S204 can refer to Step S101 - Step S104 in the corresponding embodiments above, which will not be elaborated here. Figure 2 The steps S101 - S104 in the corresponding embodiments will not be elaborated here.

[0113] Step S205: Determine the true labels corresponding to the open-set sample data.

[0114] Among them, the above multi-granularity pseudo-labels benefit from the structure information obtained after being trained with the true labels. The true labels can be used for training to classify open-set samples and closed-set samples, that is, to obtain open-set samples and closed-set samples. In this process, considering that the amount of open-set sample data is large and its corresponding sequence structure is relatively complex, therefore, the open-set samples can be decomposed by a multi-granularity decomposer to obtain multiple multi-granularity subclasses. Furthermore, the classification of open-set samples and closed-set samples can be achieved through multiple multi-granularity subclasses.

[0115] Step S206: Obtain a preset cross-entropy loss function, and train the classification task corresponding to the true labels and the multi-label classification task corresponding to the multi-granularity pseudo-labels according to the preset cross-entropy loss function to obtain the prediction results corresponding to the true labels and the prediction results corresponding to the multi-granularity labels.

[0116] Step S207: Optimize the preset cross-entropy loss function according to the prediction results corresponding to the true labels and the prediction results corresponding to the multi-granularity labels.

[0117] Among them, the classification task corresponding to the above true labels can refer to the original problem, that is, to classify and identify open-set samples and closed-set samples, that is, to separate open-set samples and closed-set samples. The multi-label classification task corresponding to the above multi-granularity pseudo-labels refers to open-set structure classification, that is, to classify open-set structure information, that is, to assign multi-granularity pseudo-labels to the data in open-set samples, and in the process of multiple update iterations, update the multi-granularity pseudo-labels.

[0118] Specifically, such as Figure 1eAs shown in the figure, it is a schematic diagram of the architecture of the sequence structure learner. When the multi-granularity pseudo-labels are trained for the t-th round through the sequence structure learner, given the training samples corresponding to the target features, according to the true label y corresponding to the true samples, the multi-granularity pseudo-labels, and the multi-granularity pseudo-labels after training Optimize the following preset cross-entropy loss function:

[0119]

[0120] where XE is the cross-entropy loss function, is the prediction result of the true label, P i (t) is the prediction result of the i-th multi-granularity pseudo-label, and λ is the hyperparameter that weighs the two loss functions.

[0121] Among them, the above preset cross-entropy loss function can be used to classify open-set samples and closed-set samples.

[0122] Among them, the first term of the above loss function is used to train the structural information obtained from the true label, and this part is the training without adding multi-granularity pseudo-labels; the second term shows that multi-granularity pseudo-labels are added to the model, which is the prediction result of the multi-granularity pseudo-labels.

[0123] Among them, when adding multi-granularity pseudo-labels, that is, using open-set sample data, the distinction between open-set samples and closed-set samples is realized, and the difficulty of the open-set recognition task is reduced; and the finally obtained model is also more prominent than the model that only includes closed-set data in the training set.

[0124] In the embodiments of the present application, open-set sample data and closed-set sample data are obtained; open-set features corresponding to the open-set sample data are extracted, and closed-set features corresponding to the closed-set sample data are extracted, and the open-set features and the closed-set features are combined into target features; further, the open-set features are input into a multi-granularity decomposer to obtain multi-granularity subclasses; furthermore, matching processing is performed on the multi-granularity subclasses to obtain multi-granularity pseudo-labels; and open-set structure information corresponding to the multi-granularity subclasses is determined; through a sequence structure learner, the target features and the multi-granularity pseudo-labels are trained to obtain a decomposition result; finally, the open-set structure information is updated according to the decomposition result; then the true label corresponding to the open-set sample data is determined; a preset cross-entropy loss function is obtained, and the classification task corresponding to the true label and the multi-label classification task corresponding to the multi-granularity pseudo-labels are trained according to the preset cross-entropy loss function to obtain a prediction result corresponding to the true label and a prediction result corresponding to the multi-granularity labels; according to the prediction result corresponding to the true label and the prediction result corresponding to the multi-granularity labels, the preset cross-entropy loss function is optimized. Therefore, by using the trained multi-granularity pseudo-labels and the trained multi-granularity pseudo-labels to optimize the preset cross-entropy loss function, the optimization of the preset cross-entropy loss function can be achieved; and since the preset cross-entropy loss function can be used to classify open-set samples and closed-set samples, therefore, easily collectable and ubiquitous open-set samples can be utilized to reduce the difficulty of the open-set sample classification task and improve the data utilization rate; and through the sequence structure learner, the target features including the features corresponding to the closed-set samples are trained, the open-set samples are dynamically decomposed according to the learning progress of the actual closed-set samples, the visual knowledge of the closed-set samples is transferred to the open-set samples, and at the same time, the subdivision of the open-set samples is used to further improve the model's recognition ability for open-closed set samples.

[0125] Please refer to Figure 3b , which is a schematic diagram of the scenario of an open-set sample decomposition method provided by the embodiments of the present application. As Figure 3bAs shown, in this scenario, the convolutional neural network is connected to the multi-granularity decomposer and the sequence structure learner. Open-set sample data and closed-set sample data can be obtained. It can be seen from the figure that there is more open-set sample data than closed-set sample data, and the corresponding categories are also more. The closed-set sample data can include pictures of cats and dogs of different breeds or poses. In addition to pictures of cats and dogs, the open-set sample data can also include pictures of rabbits, chickens, horses, etc. The open-set sample data can be processed by the convolutional neural network to obtain open-set features, and the closed-set features corresponding to the closed-set sample data can be obtained. The above open-set features and closed-set features are used as target features. The open-set features can be input into the multi-granularity decomposer, and the multi-granularity decomposer performs clustering processing on the open-set features to obtain multiple clusters. Each cluster can correspond to a multi-granularity subcategory. The open-set sample data corresponding to each cluster can be used as different classes for subsequent classification training. Furthermore, multiple clusters can be matched one by one to obtain multiple matching pairs, and the open-set sample data corresponding to the matching pairs can be labeled to obtain a set of multi-granularity pseudo-labels. For example, the multi-granularity pseudo-labels that can be assigned to a certain cluster can be: cats, dogs, etc., which are not limited here; an approximate label can be obtained according to the label data (closed-set sample data), and this approximate label is the multi-granularity pseudo-label.

[0126] Furthermore, the above set of multi-granularity pseudo-labels can be input into the sequence structure learner. It can be seen from the architecture of the sequence structure learner that there are multiple label spaces corresponding in the sequence structure learner, and the sequence structure learner embeds the target features into each label space through a fully connected layer; in addition, a set of multi-granularity pseudo-labels is input into each label space through a preset encoder to train the above set of multi-granularity pseudo-labels through the label space; during the training, since the target features are added, and the target features include the closed-set features, therefore, it is equivalent to transferring the visual knowledge of the closed-set sample data to the entire training process. After training the sequence structure learner for multiple rounds, multiple sets of multi-granularity pseudo-labels after training can be obtained, and they are combined with the set of multi-granularity pseudo-labels output by the multi-granularity decomposer to form the latest open-set structure information of the above open-set sample data; finally, the new open-set sample data corresponding to the open-set structure information can be decomposed by the multi-granularity decomposer again to obtain a set of multi-granularity pseudo-labels, and the above operation steps are continued to be repeated, which can gradually improve the recognition ability of the entire model architecture for the open-set sample data and is beneficial to improving the utilization rate of data.

[0127] In addition, by Figure 3bAs shown, in the sequence structure learner, before training with multi-dimensional pseudo-labels, the label data can be trained by the sequence structure learner, and the label data can be obtained by training with real data. The two cross-entropies in the figure may not be the same. In the embodiment of the present application, after obtaining a set of multi-granularity pseudo-labels corresponding to the open-set samples, the original classification and the open-set structure classification can be jointly trained by cross-entropy as a multi-label classification task. Among them, the above-mentioned original classification can be used to identify and classify the open-set samples and the closed-set samples through real labels. In this way, the classification and recognition of both the closed-set samples and the open-set samples can be achieved simultaneously, which is beneficial to reducing the difficulty of the open-set recognition task.

[0128] Further, please refer to Figure 4 , which is a schematic structural diagram of an open-set sample decomposition device provided by an embodiment of the present application. As Figure 4 shown, the open-set sample decomposition device 1 may include: a first acquisition module 11, a matching module 12, a training module 13, and an updating module 14;

[0129] The first acquisition module 11 is configured to acquire open-set sample data and closed-set sample data; extract the open-set features corresponding to the open-set sample data, and extract the closed-set features corresponding to the closed-set sample data, and combine the open-set features and the closed-set features into target features;

[0130] The matching module 12 is configured to input the open-set features into a multi-granularity decomposer to obtain multi-granularity subclasses; perform matching processing on the multi-granularity subclasses to obtain multi-granularity pseudo-labels; and determine the open-set structure information corresponding to the multi-granularity subclasses;

[0131] The training module 13 is configured to train the target features and the multi-granularity pseudo-labels through a sequence structure learner to obtain a decomposition result;

[0132] The updating module 14 is configured to update the open-set structure information according to the decomposition result.

[0133] Among them, the specific functional implementation manners of the first acquisition module 11, the matching module 12, the training module 13, and the updating module 14 can refer to steps S101-S104 in the corresponding embodiment above, and will not be elaborated here. Figure 2

[0134] Please refer to Figure 4 simultaneously. The matching module 12 may include: a clustering unit 121, configured to cluster the open-set features through the multi-granularity decomposer at a preset step size to obtain a plurality of clusters, and the plurality of clusters form the multi-granularity subclasses;

[0135] Figure 4 ​​, the clustering unit 121 is specifically configured to, if the outlier features are clustered twice with the preset step size, obtain a first cluster result and a second cluster result, where the first cluster result includes a plurality of first clusters, the second cluster result includes a plurality of second clusters, and the plurality of first clusters and the plurality of second clusters form the plurality of clusters.

[0136] Among them, the specific functional implementation manner of the clustering unit 121 can be referred to in the above Figure 2 corresponding step S102 in the embodiment will not be elaborated here.

[0137] Please refer to Figure 4 together, the matching module 12 may further include: a matching unit 122 and a tagging unit 123;

[0138] The matching unit 122 is configured to match the plurality of first clusters with the plurality of second clusters one by one to obtain multiple pairs of matching clusters, where the matching clusters are composed of a first cluster and a second cluster;

[0139] The tagging unit 123 is configured to tag the sample data corresponding to the sample features in the multiple pairs of matching clusters with pseudo-labels to obtain the multi-granularity pseudo-labels.

[0140] Among them, the specific functional implementation manners of the matching unit 122 and the tagging unit 123 can be referred to in the above Figure 2 corresponding step S102 in the embodiment will not be elaborated here.

[0141] Please refer to Figure 4 together, the matching unit 122 may include: a first determination subunit 1221 and a matching subunit 1222;

[0142] The first determination subunit 1221 is configured to, if there are common samples between the first cluster i in the plurality of first clusters and the second cluster j in the plurality of second clusters, and the second cluster j has the most common samples with the first cluster i compared to other second clusters except the second cluster j, determine that the first cluster i and the second cluster j are a pair of matching clusters, where the first cluster i is the i-th first cluster in the plurality of clusters, and the second cluster j is the j-th second cluster in the plurality of second clusters;

[0143] The matching subunit 1222 is configured to, after determining that the first cluster i and the second cluster j are a pair of matching clusters, continue to match the first cluster i + 1 with other second clusters except the second cluster j to obtain multiple pairs of matching clusters.

[0144] Among them, the specific functional implementation manners of the first determination subunit 1221 and the matching subunit 1222 can be referred to in the aboveFigure 2 The steps in the corresponding embodiments for S102 will not be elaborated here.

[0145] Please refer to Figure 4 , the training module 13 includes: an embedding unit 131 and a training unit 132;

[0146] The embedding unit 131 is configured to embed the target feature into multiple label spaces of the sequence structure learner through a multi-label embedder;

[0147] The training unit 132 is configured to train the multi-granularity pseudo-labels through the multiple label spaces to obtain a decomposition result composed of the multi-granularity pseudo-labels and multiple trained multi-granularity pseudo-labels.

[0148] Among them, the specific functional implementation manners of the embedding unit 131 and the training unit 132 can be referred to the above Figure 2 steps S103 in the corresponding embodiments.

[0149] Please refer to Figure 4 , the embedding unit 131 may include: a second determination subunit 1311, a third determination subunit 1312, and an embedding subunit 1313;

[0150] The second determination subunit 1311 is configured to determine the quantity corresponding to the multi-granularity pseudo-labels;

[0151] The third determination subunit 1312 is configured to determine the number of layers of the fully connected layer according to the quantity to obtain a multi-layer fully connected layer, where the number of layers of the fully connected layer corresponds to the number of label spaces;

[0152] The embedding subunit 1313 is configured to respectively embed the target feature into multiple label spaces of the sequence structure learner through the multi-layer fully connected layer and a preset encoder.

[0153] Among them, the specific functional implementation manners of the second determination subunit 1311, the third determination subunit 1312, and the embedding subunit 1313 can be referred to the above Figure 2 steps S203 in the corresponding embodiments, which will not be elaborated here.

[0154] As Figure 4 shown, the open-set sample decomposition device 1 may further include: a determination module 15, a second acquisition module 16, and an optimization module 17;

[0155] The determination module 15 is configured to determine the true labels corresponding to the open-set sample data;

[0156] The second acquisition module 16 is configured to acquire a preset cross-entropy loss function, and train a classification task corresponding to the true label and a multi-label classification task corresponding to the multi-granularity pseudo-label according to the preset cross-entropy loss function, so as to obtain a prediction result corresponding to the true label and a prediction result corresponding to the multi-granularity label;

[0157] The optimization module 17 is configured to optimize the preset cross-entropy loss function according to the prediction result corresponding to the true label and the prediction result corresponding to the multi-granularity label.

[0158] Among them, for the specific functional implementation manners of the determination module 15, the second acquisition module 16, and the optimization module 17, reference may be made to steps S305-S307 in the corresponding embodiment of FIG. 3 above, which will not be elaborated here.

[0159] In the embodiment of the present application, open-set sample data and closed-set sample data are acquired; open-set features corresponding to the open-set sample data are extracted, and closed-set features corresponding to the closed-set sample data are extracted, and the open-set features are combined with the closed-set features to form target features; furthermore, the open-set features are input into a multi-granularity decomposer to obtain multi-granularity subclasses; moreover, matching processing is performed on the multi-granularity subclasses to obtain multi-granularity pseudo-labels; and open-set structure information corresponding to the multi-granularity subclasses is determined; through a sequence structure learner, the target features and the multi-granularity pseudo-labels are trained to obtain a decomposition result; finally, the open-set structure information is updated according to the decomposition result. The open-set samples can be decomposed by the multi-granularity decomposer; and through the sequence structure learner, combined with the closed-set samples, the multi-granularity pseudo-labels obtained by the multi-granularity decomposer are learned to obtain a decomposition result, that is, the open-set structure information obtained in the current multi-round training, and the open-set structure information corresponding to the above open-set samples is updated according to the trained open-set structure information; in this way, the open-set samples can be dynamically decomposed according to the learning progress of the closed-set samples, and the visual knowledge in the closed-set information can be transferred to the open-set samples. In addition, the multi-granularity decomposer can also be used to decompose the open-set samples to obtain multi-granularity subclasses, realizing the subdivision of the open-set samples, which is beneficial to improving the recognition ability of the open-set samples.

[0160] Further, please refer to Figure 5 which is a schematic structural diagram of a terminal device provided by an embodiment of the present application. As Figure 5 shown, the above Figure 4The outlier sample decomposition device 1 in can be applied to the terminal device 500, and the terminal device 500 may include: a processor 501, a network interface 504, and a memory 505. In addition, the terminal device 500 may further include: a user interface 503 and at least one communication bus 502. Among them, the communication bus 502 is used to realize the connection and communication between these components. Among them, the user interface 503 may include a display screen (Display) and a keyboard (Keyboard). Optionally, the user interface 503 may further include a standard wired interface and a wireless interface. The network interface 504 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 504 may be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The memory 504 may optionally be at least one storage device located far from the aforementioned processor 501. As Figure 5 shown, the memory 504 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a device control application program.

[0161] In Figure 5 In the terminal device 500 shown, the network interface 504 can provide network communication functions; while the user interface 503 is mainly used to provide an input interface for users; and the processor 501 can be used to call the device control application program stored in the memory 504 to achieve:

[0162] Obtain outlier sample data and closed-set sample data; extract the outlier features corresponding to the outlier sample data, and extract the closed-set features corresponding to the closed-set sample data, and combine the outlier features and the closed-set features into target features;

[0163] Input the outlier features into a multi-granularity decomposer to obtain multi-granularity subclasses; perform matching processing on the multi-granularity subclasses to obtain multi-granularity pseudo-labels; and determine the outlier structure information corresponding to the multi-granularity subclasses;

[0164] Through a sequence structure learner, train the target features and the multi-granularity pseudo-labels to obtain a decomposition result;

[0165] Update the outlier structure information according to the decomposition result.

[0166] In one embodiment, when the processor 501 executes the step of inputting the outlier features into a multi-granularity decomposer to obtain multi-granularity subclasses, the following steps are specifically executed:

[0167] Cluster the outlier features with a preset step size through the multi-granularity decomposer to obtain a plurality of clusters, and the plurality of clusters form the multi-granularity subclasses;

[0168] Performing clustering on the open-set features with a preset step size to obtain multiple clusters, including:

[0169] If the open-set features are clustered twice with the preset step size to obtain a first cluster result and a second cluster result, the first cluster result includes multiple first clusters, the second cluster result includes multiple second clusters, and the multiple first clusters and the multiple second clusters form the multiple clusters.

[0170] In one embodiment, when the processor 501 performs the matching process on the multi-granularity subclasses to obtain multi-granularity pseudo-labels, the following steps are specifically executed:

[0171] One-to-one matching the multiple first clusters with the multiple second clusters to obtain multiple pairs of matching clusters, where the matching clusters are composed of a first cluster and a second cluster;

[0172] Assigning pseudo-labels to the sample data corresponding to the sample features in the multiple pairs of matching clusters to obtain the multi-granularity pseudo-labels.

[0173] In one embodiment, when the processor 501 performs the one-to-one matching of the multiple first clusters with the multiple second clusters, the following steps are specifically executed:

[0174] If there are common samples between the first cluster i in the multiple first clusters and the second cluster j in the multiple second clusters, and the second cluster j has the most common samples with the first cluster i compared to other second clusters except the second cluster j, then determine that the first cluster i and the second cluster j are a pair of matching clusters, where the first cluster i is the i-th first cluster in the multiple clusters, and the second cluster j is the j-th second cluster in the multiple second clusters;

[0175] After determining that the first cluster i and the second cluster j are a pair of matching clusters, continue to match the first cluster i + 1 with other second clusters except the second cluster j to obtain multiple pairs of matching clusters.

[0176] In one embodiment, when the processor 501 performs training on the target feature and the multi-granularity pseudo-labels through a sequence structure learner to obtain a decomposition result, the following steps are specifically executed:

[0177] Embedding the target feature into multiple label spaces of the sequence structure learner through a multi-label embedder;

[0178] Training the multi-granularity pseudo-labels through the multiple label spaces to obtain a decomposition result composed of the multi-granularity pseudo-labels and multiple trained multi-granularity pseudo-labels.

[0179] In one embodiment, when the processor 501 executes embedding the target feature into multiple label spaces of the sequence structure learner through the multi-label embedder, the following steps are specifically executed:

[0180] Determine the quantity corresponding to the multi-granularity pseudo-labels;

[0181] According to the quantity, determine the number of layers of the fully connected layer to obtain a fully connected layer with a multi-layer structure, where the number of layers of the fully connected layer corresponds to the number of label spaces;

[0182] Embed the target feature into multiple label spaces of the sequence structure learner through the multi-layer structure fully connected layer and a preset encoder.

[0183] In one embodiment, the processor 501 further executes the following steps:

[0184] Determine the true labels corresponding to the open-set sample data;

[0185] Obtain a preset cross-entropy loss function, and train the classification task corresponding to the true labels and the multi-label classification task corresponding to the multi-granularity pseudo-labels according to the preset cross-entropy loss function to obtain the prediction results corresponding to the true labels and the prediction results corresponding to the multi-granularity labels;

[0186] Optimize the preset cross-entropy loss function according to the prediction results corresponding to the true labels and the prediction results corresponding to the multi-granularity labels.

[0187] In an embodiment of the present application, open-set sample data and closed-set sample data are obtained; open-set features corresponding to the open-set sample data are extracted, and closed-set features corresponding to the closed-set sample data are extracted, and the open-set features and the closed-set features are combined into target features; furthermore, the open-set features are input into a multi-granularity decomposer to obtain multi-granularity subclasses; moreover, matching processing is performed on the multi-granularity subclasses to obtain multi-granularity pseudo-labels; and open-set structure information corresponding to the multi-granularity subclasses is determined; through a sequence structure learner, the target features and the multi-granularity pseudo-labels are trained to obtain a decomposition result; finally, the open-set structure information is updated according to the decomposition result. The open-set samples can be decomposed by the multi-granularity decomposer; and through the sequence structure learner, combined with the closed-set samples, the multi-granularity pseudo-labels obtained by the multi-granularity decomposer are learned to obtain a decomposition result, that is, the open-set structure information obtained in the current multi-round training, and the open-set structure information corresponding to the above open-set samples is updated according to the trained open-set structure information; in this way, by cycling, the open-set samples can be dynamically decomposed according to the learning progress of the closed-set samples, and the visual knowledge in the closed-set information can be transferred to the open-set samples. In addition, the multi-granularity decomposer can also be used to decompose the open-set samples to obtain multi-granularity subclasses, realizing the subdivision of the open-set samples, which is beneficial to improving the recognition ability of the open-set samples.

[0188] It should be understood that the terminal device 500 described in the embodiments of the present application can execute the description of the open-set sample decomposition method in the corresponding embodiments mentioned above Figures 2 to 3a and can also execute the description of the open-set sample decomposition device in the corresponding embodiments mentioned above Figure 4 and will not be elaborated here. In addition, the description of the beneficial effects of adopting the same method will not be elaborated either.

[0189] An embodiment of the present application provides a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the description of the open-set sample decomposition method in the corresponding embodiments mentioned above Figures 2 to 3a and will not be elaborated here.

[0190] In addition, it should be pointed out here that: an embodiment of the present application also provides a computer storage medium, and the computer storage medium stores the computer program executed by the open-set sample decomposition device 1 mentioned above, and the computer program includes program instructions. When the processor executes the program instructions, it can execute the above Figures 2 to 3aThe description of the open set sample decomposition method in the corresponding embodiments will not be repeated here. In addition, the description of the beneficial effects of using the same method will not be repeated. For the technical details not disclosed in the embodiments of the computer storage medium involved in this application, please refer to the description of the method embodiments of this application.

[0191] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc.

[0192] The above-disclosed are only the preferred embodiments of this application. Of course, the scope of rights of this application cannot be limited thereby. Therefore, equivalent changes made according to the claims of this application still fall within the scope covered by this application.

Claims

1. An open set sample decomposition method, characterized in that, Including: Obtaining open-set sample data and closed-set sample data; The open-set sample data includes images of known categories and images of unknown categories, and the images in the closed-set sample data are all images of known categories; Extracting open-set features corresponding to the images in the open-set sample data, and extracting closed-set features corresponding to the images in the closed-set sample data, and combining the open-set features and the closed-set features into target features; Inputting the open-set features into a multi-granularity decomposer to obtain multi-granularity subclasses; Performing matching processing on the multi-granularity subclasses to obtain multi-granularity pseudo-labels; And determining open-set structure information corresponding to the multi-granularity subclasses; Embedding the target features into multiple label spaces of a sequence structure learner through a multi-label embedder; the sequence structure learner includes a preset encoder, multiple label spaces, and a fully connected layer with a multi-layer structure, the number of layers of the fully connected layer corresponds to the number of the label spaces, and the fully connected layer with the multi-layer structure and the preset encoder are used to embed the target features into the multiple label spaces of the sequence structure learner respectively; Training the multi-granularity pseudo-labels through the multiple label spaces to obtain a decomposition result composed of the multi-granularity pseudo-labels and multiple trained multi-granularity pseudo-labels; Updating the open-set structure information according to the decomposition result.

2. The method according to claim 1, wherein The step of inputting the open-set features into a multi-granularity decomposer to obtain multi-granularity subclasses includes: Clustering the open-set features by the multi-granularity decomposer at a preset step size to obtain multiple clusters, and the multiple clusters form the multi-granularity subclasses; The step of clustering the open-set features at a preset step size to obtain multiple clusters includes: If the open-set features are clustered twice at the preset step size to obtain a first cluster result and a second cluster result, the first cluster result includes multiple first clusters, the second cluster result includes multiple second clusters, and the multiple first clusters and the multiple second clusters form the multiple clusters.

3. The method according to claim 2, characterized in that, The step of performing matching processing on the multi-granularity subclasses to obtain multi-granularity pseudo-labels includes: Performing one-to-one matching between the multiple first clusters and the multiple second clusters to obtain multiple pairs of matching clusters, where the matching clusters are composed of a first cluster and a second cluster; Labeling the sample data corresponding to the sample features in the multiple pairs of matching clusters with pseudo-labels to obtain the multi-granularity pseudo-labels.

4. The method according to claim 3, characterized in that, The step of performing one-to-one matching between the multiple first clusters and the multiple second clusters includes: If there are common samples between the first cluster i in the multiple first clusters and the second cluster j in the multiple second clusters, and compared with other second clusters except the second cluster j, the second cluster j has the most common samples with the first cluster i, then determining the first cluster i and the second cluster j as a pair of matching clusters, where the first cluster i is the i-th first cluster in the multiple clusters, and the second cluster j is the j-th second cluster in the multiple second clusters; After determining that the first type of cluster \(i\) and the second type of cluster \(j\) are a pair of matching clusters, continue to match the first type of cluster \(i + 1\) with other second type of clusters except the second type of cluster \(j\) to obtain multiple pairs of matching clusters.

5. The method according to claim 1, characterized in that, The embedding of the target feature into multiple label spaces of the sequence structure learner by the multi-label embedder includes: Determine the number corresponding to the multi-granularity pseudo-label; According to the number, determine the number of layers of the fully connected layer to obtain a fully connected layer with a multi-layer structure, where the number of layers of the fully connected layer corresponds to the number of label spaces; Embed the target feature into multiple label spaces of the sequence structure learner through the fully connected layer with the multi-layer structure and a preset encoder respectively.

6. The method according to claim 1, characterized in that, The method further includes: Determine the true label corresponding to the open-set sample data; Obtain a preset cross-entropy loss function, and train the classification task corresponding to the true label and the multi-label classification task corresponding to the multi-granularity pseudo-label according to the preset cross-entropy loss function to obtain the prediction result corresponding to the true label and the prediction result corresponding to the multi-granularity label; Optimize the preset cross-entropy loss function according to the prediction result corresponding to the true label and the prediction result corresponding to the multi-granularity label.

7. An open set sample decomposition device, characterized in that, Includes: A first acquisition module for acquiring open-set sample data and closed-set sample data; Extract the open-set feature corresponding to the image in the open-set sample data, and extract the closed-set feature corresponding to the image in the closed-set sample data, and combine the open-set feature and the closed-set feature into a target feature; The open-set sample data includes images of known categories and images of unknown categories, and the images in the closed-set sample data are all images of known categories; A matching module for inputting the open-set feature into a multi-granularity decomposer to obtain multi-granularity subclasses; Perform matching processing on the multi-granularity subclasses to obtain multi-granularity pseudo-labels; And determine the open-set structure information corresponding to the multi-granularity subclasses; A training module for embedding the target feature into multiple label spaces of a sequence structure learner through a multi-label embedder, and training the multi-granularity pseudo-labels through the multiple label spaces to obtain a decomposition result composed of the multi-granularity pseudo-labels and multiple trained multi-granularity pseudo-labels; The sequence structure learner includes a preset encoder, multiple label spaces and a fully connected layer with a multi-layer structure, the number of layers of the fully connected layer corresponds to the number of label spaces, and the fully connected layer with the multi-layer structure and the preset encoder are used to embed the target feature into multiple label spaces of the sequence structure learner respectively; An updating module for updating the open-set structure information according to the decomposition result.

8. A terminal device, characterized in that, Includes: A processor and a memory; The processor and the memory are connected, where the memory is used to store program codes, and the processor is used to call the program codes to execute the method according to any one of claims 1-6.

9. A computer storage medium, characterized in that, The computer storage medium stores a computer program, and the computer program includes program instructions, and when the program instructions are executed by a processor, the method according to any one of claims 1-6 is executed.

10. A computer program product, the computer program product comprising computer instructions, the computer instructions being stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method according to any one of claims 1-6.

Images