Picture classification management method based on semi-supervised scene recognition

Abstract

The invention discloses a picture classification management method based on semi-supervised scene recognition, and the method comprises the steps: introducing a vision-language cross-modal model, and defining a labeled picture set and an unlabeled picture set as a model training set; extracting the visual features of the picture and the features of the text annotation of the picture; extracting global information which is embedded after pooling of the visual feature map and serves as a picture sample, and inputting the global information into a scene discriminator to obtain a scene discrimination result; embedding before pooling of the visual feature map is input, and category-related features are calculated; dividing a label space into a plurality of clusters by utilizing a co-occurrence relationship of categories in the picture data set; and meanwhile, an auxiliary task of scene recognition is introduced, classes are divided into different clusters by utilizing co-occurrence information in the labels, and global features of the picture are input to discriminate the scene to obtain a loss function. According to the method, the picture classification speed and precision can be improved, and meanwhile, a scene keyword retrieval function is provided to help a user to quickly retrieve and position a specified picture.

Description

Technical Field

[0001] The present invention belongs to the field of picture classification, and in particular relates to a picture classification management method based on semi-supervised scene recognition. Background Art

[0002] The development of deep learning has led to the widespread application of deep learning networks in different industries, such as recommendation systems, medical diagnosis, and video and image annotation. The essential condition for training deep learning models is a large amount of accurately labeled data. However, in real scenarios, obtaining a large amount of accurately labeled data requires a lot of resources. In the scenario of image classification management, if traditional solutions are used to accurately classify and group images, a data engine must be built to complete batch labeling of data. This operation includes data collection, data cleaning, feature extraction, manual or machine-assisted labeling, labeling quality review and other complex operations. The process is complex and requires high financial support. In addition, in user album management, it is impractical to label data on a large scale due to privacy issues. Therefore, in real-life image classification management scenarios, data generally consists of a small number of accurately labeled samples and a large number of unlabeled samples, which means that limited supervisory information is needed to train the model to achieve accurate and rapid classification of images.

[0003] Therefore, in order to meet these challenges, the present invention proposes an image classification and management method based on semi-supervised scene recognition, which can efficiently utilize the supervisory information of limited labeled images provided by users, and use the paradigm of semi-supervised learning to help the model learn on unlabeled image samples to achieve image classification and management. Summary of the invention

[0004] Purpose of the invention: In order to classify massive images and further group them according to scene information to achieve fast retrieval, the present invention proposes an image classification management method based on semi-supervised scene recognition.

[0005] Technical solution: The image classification management method based on semi-supervised scene recognition described in the present invention comprises the following steps:

[0006] (1) Given an image dataset D, it contains a set of labeled images D provided by the user. l and the unlabeled image collection D u , as the visual-language cross-modal model training set, and initialize the visual-language cross-modal model to optimize the parameters;

[0007] (2) Randomly select B samples from the training set, including labeled sample images and unlabeled sample images; at the same time, use the visual encoder E in the visual-language model V (·) and the text encoder E T(·) Extract the image visual features and the features of the image text annotations respectively;

[0008] (3) Extract the embedding of the pooled visual feature map as the global information of the image sample and input it into the scene discriminator to obtain the scene identification result;

[0009] (4) Input the embedding of the visual feature map before pooling and calculate the category-related features;

[0010] (5) Using the co-occurrence relationship of categories in the image dataset, the label space is divided into several clusters, each cluster represents scene information;

[0011] (6) Combine the global image features obtained in step (3) and the scene segmentation results in step (5) to calculate the auxiliary loss L aux ;

[0012] (7) Using the category-related features extracted in step (4), calculate the supervision loss L according to whether the sample has a label. sup and the unsupervised loss L unsup ;

[0013] (8) Optimizing the total loss function and minimizing it is used as the training objective, the parameters of the optimizable prompt words are updated and the process returns to step (2) for iterative optimization, and finally the visual-language model reaches a convergence state.

[0014] Furthermore, the implementation process of step (1) is as follows:

[0015] There is a labeled image dataset of size m And an unlabeled image dataset of size n Among them, y i is the label of the image sample. Multiple elements in the label vector are set to 1, indicating that a single image contains multiple objects.

[0016] Initialize the optimizable parameters of the visual-language cross-modal model: Initialize the model classifier by constructing an optimizable cue word t consisting of category names, and input the optimizable cue word into the text encoder E in the visual-language model. T (·) Extract text features as the final category classifier.

[0017] Furthermore, the implementation process of step (2) is as follows:

[0018] Input the optimizable hint word t with the category name into the text encoder E T (·) extracts the text feature w and inputs the image into the visual encoder E V (·) extracts visual features f i ; The characteristics of the two can be expressed as:

[0019] f i =E V (x i )

[0020] w=E T (t).

[0021] Furthermore, the implementation process of step (3) is as follows:

[0022] There is an attention pooling layer in the visual branch of the vision-language model, which is located at the visual encoder E V (·) Then, the visual feature f i Inputting into the attention pooling layer AttenPool can extract the global feature g at the same time i and local features l i :

[0023] [g i ,l i ]=AttenPool(f i )

[0024] The global feature g i Input into the scene discriminator to obtain the model's predicted probability for the current scene; for the strongly enhanced sample, the predicted probability output by the discriminator is p a , the predicted probability of the weakly enhanced sample discriminator output is q a ; Then the classification result of the scene discriminator is expressed as: Indicates the scene group to which the image belongs, allowing users to quickly search for images.

[0025] Furthermore, the implementation process of step (4) is as follows:

[0026] Use the local features of the image extracted from the visual branch l i , and an optimizable hint word t consisting of the category name, using the cross attention mechanism to calculate the category information contained in the local feature map of the image; taking t as the query vector, l i As key vector and value vector, the specific calculation process is:

[0027] z i =TransFormerDec(t,l i ,l i )

[0028] Among them, z i Used to represent the parts of the image visual features that are related to different category features.

[0029] Furthermore, the implementation process of step (5) is as follows:

[0030] In a limited set of labeled image samples, the co-occurrence relationship between different categories can assist model prediction. By calculating the number of co-occurrences of different categories, a co-occurrence matrix S of size C×C can be obtained:

[0031]

[0032] Among them, C represents the number of categories, e kl Indicates the number of samples of the kth category and the lth category that appear simultaneously, n k represents the number of samples that only contain the kth category; cluster analysis is performed on the co-occurrence matrix S, and the original label space is divided into several clusters according to the co-occurrence of different categories, where each cluster can represent a separate scene; specifically, the co-occurrence matrix is ​​first used to obtain the similarity matrix P of the category co-occurrence relationship diagram:

[0033]

[0034] After the scene is divided, it is expressed as a standardized graph cutting problem:

[0035]

[0036] Where D is a diagonal matrix, and its diagonal elements can be expressed as At the same time, L = DP is the Laplacian matrix; F represents the learned embedding representations of different categories, which contains the co-occurrence information of the categories. It represents the k smallest eigenvalues; when the trace of the matrix is ​​the smallest, Using the k-means clustering operation, the original label space can be divided into K subsets. For each labeled sample, the scene c to which it belongs i The clustering results can be used to make a judgment.

[0037] Furthermore, the implementation process of step (6) is as follows:

[0038] Using the global features of the image and the label grouping results, the loss function of the scene identification auxiliary task is obtained according to the paradigm of semi-supervised learning:

[0039]

[0040] Among them, τ is the threshold set for the scene identification auxiliary task, L ce is the cross entropy loss; only when the predicted probability of the output is greater than the threshold will the model take the loss of the sample into account for optimization.

[0041] Furthermore, the implementation process of step (7) is as follows:

[0042] Using category-related features z i , and after the text encoder E T (·) The extracted features of the learnable prompt words t t , calculate the dot product of the two to get the predicted probability value of the sample in different categories:

[0043]

[0044] Among them, σ(·) is the sigmoid activation function, which is used here and To represent the category prediction probability of weakly enhanced and strongly enhanced versions of samples respectively; then use a separate threshold for each category to pseudo-label the unlabeled samples The specific calculation process is:

[0045]

[0046] in, and They are used to represent the thresholds of positive and negative samples in unlabeled samples, q jk Represents the predicted probability of the weakly enhanced sample on the kth category; in obtaining the pseudo label of the unlabeled data After that, the unsupervised loss of n unlabeled samples belonging to C categories is calculated as:

[0047]

[0048] Predict the probability p based on the picture k and label y k , define L(p k ,y k )=y k L + (p k )+(1-y k )L - (p k ) is the multi-label loss, γ1 and γ2 are used to adjust the degree of attention paid by the loss function to positive and negative samples; the supervised loss of m labeled samples belonging to C categories can be expressed as:

[0049]

[0050] Furthermore, the total loss function of step (8) is:

[0051] L=L aux +L sup +L unsup

[0052] Among them, L aux is the scene identification auxiliary task loss, L sup For the supervised loss of labeled samples, L unsup is the unsupervised loss for unlabeled samples.

[0053] Beneficial effects: Compared with the prior art, the present invention has the following beneficial effects: the present invention utilizes the rich knowledge obtained by the visual-language model in the pre-training stage to achieve refined annotation of images and improve the accuracy of image classification; during the training process, the model parameters of the visual and language branches are kept frozen, and only the optimizable prompt words are initialized for training, which greatly reduces the training cost. By utilizing the implicit knowledge in the visual-language model, the annotation cost can be significantly reduced, eliminating the need to build a cumbersome data engine, and only using limited supervision information to complete model training. Therefore, it has excellent application value and prospects in scenarios where computing resources are severely limited in real life, such as mobile phones and tablet devices. In the album, it can be used for automatic grouping and search of photos, and can be widely used in different scenarios; at the same time, based on the multimodal design, the present invention can give the text label corresponding to the picture to assist the subsequent task of grouping pictures according to the scene, and provide users with a convenient and fast retrieval function; in addition, the present invention introduces the auxiliary task of scene identification, and performs cross-attention operations between different modal features to further improve the degree of alignment between modalities, thereby reducing the dependence on a large amount of accurately labeled data, meeting the needs of actual application scenarios, so that the model completes the classification of pictures and scene grouping management only by using limited user annotation information, and improves the accuracy and speed of picture classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0055] The present invention is further described in detail below with reference to the accompanying drawings.

[0056] like Figure 1 As shown, the present invention proposes a method for image classification management based on semi-supervised scene recognition, which fine-tunes a visual-language model pre-trained on a large number of samples to achieve application in the field of image management, and specifically includes the following steps:

[0057] Step 1: Given a label set D l and the unlabeled set D u As a model training set, and initialize the model to optimize parameters.

[0058] Specifically, the training data contains a labeled image dataset And unlabeled image dataset Among them, y i For the labels of multi-label samples, multiple elements in this vector are set to 1, indicating that a single image sample contains semantic information of multiple objects. The initialization model components include: optimizable prompt words and scene discriminators. The model classifier is initialized by constructing an optimizable prompt word t consisting of category names, and the optimizable prompt word is input into the text encoder E in the visual-language model. T (·) Extract text features as the final category classifier. The input image passes through the visual encoder E in the model. V (·) After extracting the features, the category classifier can be used to obtain the model’s prediction probability for different categories.

[0059] In this embodiment, we take the image management in the mobile phone album as an example. Initially, 20,000 images are given, which contains a small number of labeled data sets that have been annotated by users. It contains 100 images with detailed label information; and most of the unlabeled dataset The number of these unlabeled pictures reached 19,900. These pictures cover various scenes in daily life, such as family, transportation, scenery and other themes. In this scenario, the goal of image management is to automatically extract and analyze the semantic information of a large amount of unsupervised data through training with a small amount of labeled data, and to classify similar pictures into different groups so that users can easily retrieve and browse image content of different categories. Specifically, each picture often contains multiple objects or scene information. For example, a photo may contain both annotations for the object class "car" and scene class annotations such as "city street" or "highway". In addition to the annotation information of different objects, each picture also contains scene label information, so the label vector y of the sample i There are multiple elements in which the value is set to 1, corresponding to the labels of different categories of information.

[0060] Step 2: Randomly select B samples from the training data set, including labeled samples and unlabeled samples. At the same time, use the visual encoder E in the visual-language model V (·) and the text encoder E T (·) Extract sample features f respectively i And the feature w of the category prompt word. The two features can be expressed as:

[0061] f i =E V (x i ), w=E T (t)

[0062] By constructing an optimizable prompt word containing a category name and inputting it into a text encoder to extract text features as the final category classifier, this method can effectively provide prior knowledge for model optimization. In this embodiment, when predicting the category to which an object in a picture belongs, the text encoder can receive any text description as category prior knowledge, so it can cover the vast label space given by users in actual scenarios. For example, if a user wants to search for the object "bus" in an album, the text encoder can also receive the keyword "bus". In addition, when constructing optimizable prompt words, model labels can also be given as abstract concepts such as joy and sadness. Such a design can serve the user's demand for abstract semantics in actual scenarios.

[0063] Step 3: Extract the embedding of the pooled visual feature map as a comprehensive understanding of the image sample and input it into the scene discriminator to obtain the scene identification result.

[0064] The scene identification information provided by the user can also assist in the annotation of objects on detailed objects. Assuming that the scene information provided is a living room, the probability of the occurrence of categories such as TV, remote control, dining table and flower pot will increase in this scene. Therefore, the scene information in the annotation information can be used to better serve the grouping of album pictures. There is an attention pooling layer in the visual branch of the visual-language model, which is located at the visual encoder E V (·) Then, the visual feature f i Inputting into the attention pooling layer can extract the global feature g at the same time i and local features l i :

[0065] [g i , l i ]=AttenPool(f i )

[0066] The global feature g i Input into the scene discriminator to get the model's predicted probability for the current scene, which represents the model's overall understanding of the image. If the input is an unlabeled sample, then for the strongly enhanced sample, the predicted probability output by the discriminator is p a , the predicted probability of the weakly enhanced sample discriminator output is q a . Then the classification result of the scene discriminator can be expressed as: final It can be used to indicate the scene to which the unlabeled sample belongs.

[0067] Step 4: Input the embedding of the visual feature map before pooling and calculate the category-related features.

[0068] In actual album management scenarios, the mechanism of calculating category-related features plays a significant role in the automatic organization and classification of large-scale images. When users upload a large number of pictures, the system can automatically extract local features from each picture and combine them with category prompts (such as "wedding", "travel", "pets", "scenery", etc.) to group and annotate the pictures in detail. For example, for a photo containing multiple scenes, the system can accurately identify multiple category features such as "crowd", "beach", "sun umbrella", etc. through the cross-attention mechanism, and generate multi-label annotations containing these key information. This not only improves the accuracy of image grouping, but also provides more dimensional detailed information in annotation, greatly reducing the burden of manual organization for users.

[0069] Specifically, the present invention uses the local features of the image extracted from the visual branch i , and an optimizable hint word t consisting of the category name, using the cross attention mechanism to calculate the category information contained in the local feature map of the image. Take t as the query vector, l i As key vector and value vector, the specific calculation process is:

[0070] z i =TransFormerDec(t,l i , l i )

[0071] z i That is, it can be used to represent the parts of the visual features of the image that are related to different category features. In the scenario of multi-label learning, this method can further improve the model's ability to recognize category features, while improving the alignment between the visual modality and the text, to ensure that the pictures and text descriptions in the album application are more consistent and coherent. In actual operation, this is manifested in the system being able to intelligently generate picture labels or annotations through text prompts to ensure that the description is closely related to the image content. For example, for a photo taken at a dinner party, the system can not only detect visual elements such as "dining table", "food", and "candlelight" in the picture, but also generate highly semantic labels such as "warm dinner" or "family reunion". This allows users to quickly locate relevant pictures by keywords or categories when searching and browsing albums, greatly improving management efficiency.

[0072] Step 5: Use the co-occurrence relationship of categories in the data set to divide the label space into several clusters, each of which can represent scene information.

[0073] In the context of album management, the co-occurrence relationship between different categories can effectively help the model make more accurate category predictions. In photos of daily life, many categories frequently appear together. For example, in photos of family gatherings, "dining table", "food" and "people" often appear together. This co-occurrence pattern between categories can provide additional information for the album system to better predict and organize images. The co-occurrence relationship between different categories can assist model prediction. By calculating the number of co-occurrences of different categories, a co-occurrence matrix S of size C×C can be obtained:

[0074]

[0075] Among them, C represents the number of categories, here e kl Indicates the number of samples of the kth category and the lth category that appear simultaneously, n k It represents the number of samples that only contain the kth category. After that, cluster analysis is performed on the co-occurrence matrix S. The original label space can be divided into several clusters according to the co-occurrence of different categories, where each cluster can represent a separate scene. For example, pictures containing "beach", "waves" and "coconut trees" can be classified as "seaside vacation" scenes, while pictures containing "people", "cakes" and "candles" may be classified as "birthday party" scenes. In this way, the system can compress and organize the original label space according to the co-occurrence of categories in the picture, thereby improving the efficiency of picture management and classification. Specifically, the co-occurrence matrix is ​​first used to obtain the similarity matrix P of the category co-occurrence relationship graph:

[0076]

[0077] Scene partitioning can then be formulated as a normalized graph cut problem:

[0078]

[0079] Where D is a diagonal matrix, and its diagonal elements can be expressed as At the same time, L = DP is the Laplacian matrix. F represents the learned embedding representations of different categories, which contains the co-occurrence information of the categories. It represents the k smallest eigenvalues. Using the k-means clustering operation, the original label space can be divided into K subsets. For each labeled sample, the scene c to which it belongs i The clustering results can be used to make a judgment.

[0080] Step 6: Calculate the auxiliary loss L by combining the global image features obtained in step 3 and the scene segmentation results in step 5 aux .

[0081] Using the global features of the image and the label grouping results obtained based on category co-occurrence analysis, the system can further design scene identification auxiliary tasks to improve the ability to recognize image categories and scenes. By integrating this task into the semi-supervised learning framework, a large amount of unlabeled data can be effectively utilized to improve the generalization ability and classification performance of the model. Assuming that the labeled set has m samples and the unlabeled set has n samples, the loss function is:

[0082]

[0083] Among them, τ is the threshold set for the scene identification auxiliary task. Only when the output prediction probability is greater than the threshold, the model will take the loss of the sample into account for optimization. ce The cross entropy loss is used to measure the difference between the probability distribution predicted by the model and the actual scene label. By minimizing this part of the loss, the model can gradually learn how to use global features and label grouping results to accurately identify different scenes, which provides a more intelligent and sophisticated method for organizing and classifying pictures for the album management system.

[0084] Step 7: Use the category-related features extracted in step 4 to calculate the supervision loss L according to whether the sample has a label. sup and the unsupervised loss L unsup .

[0085] Using the obtained category-related features z i , and after the text encoder E T (·) The extracted features of the learnable prompt words t t , calculate the dot product of the two to get the predicted probability value of the sample in different categories

[0086]

[0087] Where σ(·) is the sigmoid activation function, used here and To represent the category prediction probabilities of weakly enhanced and strongly enhanced versions of samples, respectively. Then, a separate threshold for each category is used to pseudo-label the unlabeled samples. The specific calculation process is:

[0088]

[0089] in and They are used to represent the thresholds of positive and negative samples in unlabeled samples, q jk Represents the predicted value on the kth category. Afterwards, the unsupervised loss can be calculated as:

[0090]

[0091] Predict the probability p based on the picture k and the true label y k , define L(p k ,y k )=y k L + (p k )+(1-y k )L - (p k ) is the multi-label loss, where γ1 and γ2 can be used to adjust the degree of attention paid by the loss function to positive and negative samples. The supervised loss can also be expressed as:

[0092]

[0093] In step 8, the total loss function is optimized and minimized as the training objective, the parameters of the optimizable prompt words are updated and the optimization is performed back to step 2 for iterative optimization, and finally the model reaches a convergence state.

[0094] L=L aux +L sup +L unsup

[0095] Among them, L aux is the scene identification auxiliary task loss, L sup For the supervised loss of labeled samples, L unsup is the unsupervised loss of unlabeled samples. According to the above steps, the model can complete the first round of optimization. After that, the model can be iteratively optimized according to the same steps until convergence, and finally the optimized prompt words can be used as the final classifier of the model. In actual application scenarios, since the parameters to be optimized only exist in a small number of prompt words, the high computing power requirements and time overhead are saved. After a new picture appears in the album, the user can quickly get the category label of the new picture and the scene group to which it belongs.

[0096] The present invention is particularly suitable for large-scale image management systems, especially in the album application scenarios of mobile devices. As the number of pictures stored in the user's mobile phone album increases rapidly, it becomes extremely difficult to manually classify and manage pictures. Through the intelligent picture classification method based on category co-occurrence analysis and semi-supervised learning proposed in the present invention, users can automatically and efficiently group massive pictures by scene and category, reducing the tedious operation of manual sorting. The method of extracting category-related features in samples proposed in the present invention can enhance the interaction between different modalities. At the same time, the auxiliary task of scene identification not only improves the prediction accuracy of the model, but also greatly helps users in the actual needs of using albums to manage pictures.

[0097] In order to verify the superiority of this method, this implementation method evaluates the model on two commonly used multi-label image datasets: MS-COCO 2014 (COCO) and VOC-2012 (VOC). These two datasets cover a large number of pictures commonly seen in daily life and are used to simulate the user's needs for fine-grained annotation and scene grouping of a large number of pictures. Existing methods in the field of semi-supervised multi-label are collected for comparison. Among the comparison methods, BCE and ASL are commonly used loss functions in multi-label learning; DualCoOp, SCPNet, TaI and DualCoOp++ are also methods based on fine-tuning the visual-language model.

[0098] COCO contains 82,081 training set images and 40,137 test set images, distributed in 80 different categories. VOC contains 5,717 training images and 5,823 test images, covering 20 categories. In order to simulate the different proportions of labeled and unlabeled data in real album management scenarios, the proportion of labeled data is set to p∈{0.05, 0.1, 0.15, 0.20}. The evaluation indicator is the mean average precision (mAP) on the test set.

[0099] ResNet-50 was used as the image encoder backbone network of the model, and the Transformer of the visual-language model was used as the text encoder. The parameters of these two backbone networks remained frozen throughout the training process. A category-specific prompt word setting was adopted, that is, each category had an independent set of parameters. Two independent prompt word vectors were optimized, and the length of the optimizable prompt word was set to 16. In order to generate category-specific features, a Transformer decoder consisting of 2 layers and 4 attention heads was used. The spectral clustering method was used to divide the original label space into K = 6 clusters (for COCO) and K = 2 clusters (for VOC). The optimizer used the AdamW optimizer, and the learning rate was set to 0.001. The threshold for scene identification was uniformly set to 0.9. The experimental results are shown in Table 1:

[0100] Table 1 mAP of different methods on different datasets

[0101]

[0102] The present invention achieves the best performance in all experimental settings, especially in the case of small label ratios, significantly surpassing other methods without incurring excessive computational costs. The performance improvement is mainly attributed to the introduction of the auxiliary task of scene identification and the better feature representation obtained by extracting category-related features in the image through the semantic-aware alignment target task. The present invention is suitable for a variety of practical application scenarios without introducing too much overhead, especially in large-scale image and video management systems, where users can automatically classify and annotate massive photos, making it convenient for users to quickly find photos of specific scenes based on text tags, such as family gatherings, travel, and landscape photos.

[0103] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for image classification management based on semi-supervised scene recognition, characterized in that: The following steps are involved: (1) Given an image dataset D, it contains a set of labeled images D provided by the user. l and the unlabeled image collection D u , as the visual-language cross-modal model training set, and initialize the visual-language cross-modal model to optimize the parameters; (2) Randomly select B samples from the training set, including labeled sample images and unlabeled sample images; at the same time, use the visual encoder E in the visual-language model V (·) and the text encoder E T (·) Extract the image visual features and the features of the image text annotations respectively; (3) Extract the embedding of the pooled visual feature map as the global information of the image sample and input it into the scene discriminator to obtain the scene identification result; (4) Input the embedding of the visual feature map before pooling and calculate the category-related features; (5) Using the co-occurrence relationship of categories in the image dataset, the label space is divided into several clusters, each cluster represents scene information; (6) Combine the global image features obtained in step (3) and the scene segmentation results in step (5) to calculate the auxiliary loss L aux ; (7) Using the category-related features extracted in step (4), calculate the supervision loss L according to whether the sample has a label. sup and the unsupervised loss L unsup ; (8) Optimizing the total loss function and minimizing it is used as the training objective, the parameters of the optimizable prompt words are updated and the process returns to step (2) for iterative optimization, and finally the visual-language model reaches a convergence state.

2. According to the method of claim 1, the image classification management method based on semi-supervised scene recognition is characterized in that: The implementation process of step (1) is as follows: There is a labeled image dataset of size m And an unlabeled image dataset of size n Among them, y i is the label of the image sample. Multiple elements in the label vector are set to 1, indicating that a single image contains multiple objects. Initialize the optimizable parameters of the visual-language cross-modal model: Initialize the model classifier by constructing an optimizable cue word t consisting of category names, and input the optimizable cue word into the text encoder E in the visual-language model. T (·) Extract text features as the final category classifier.

3. According to the method of claim 1, the image classification management method based on semi-supervised scene recognition is characterized in that: The implementation process of step (2) is as follows: Input the optimizable hint word t with the category name into the text encoder E T (·) extracts the text feature w and inputs the image into the visual encoder E V (·) extracts visual features f i ; The characteristics of the two can be expressed as: f i =E V (x i ) w=E T (t)。 4. The image classification management method based on semi-supervised scene recognition according to claim 1 is characterized in that: The implementation process of step (3) is as follows: There is an attention pooling layer in the visual branch of the vision-language model, which is located at the visual encoder E V (·) Then, the visual feature f i Inputting into the attention pooling layer AttenPool can extract the global feature g at the same time i and local features l i : [g i ,l i ]=AttenPool(f i ) The global feature g i Input into the scene discriminator to obtain the model's predicted probability for the current scene; for the strongly enhanced sample, the predicted probability output by the discriminator is p a , the predicted probability of the weakly enhanced sample discriminator output is q a ; Then the classification result of the scene discriminator is expressed as: Indicates the scene group to which the image belongs, allowing users to quickly search for images.

5. The image classification management method based on semi-supervised scene recognition according to claim 1 is characterized in that: The implementation process of step (4) is as follows: Use the local features of the image extracted from the visual branch l i , and an optimizable hint word t consisting of the category name, using the cross attention mechanism to calculate the category information contained in the local feature map of the image; taking t as the query vector, l i As key vector and value vector, the specific calculation process is: z i =TransFormerDec(t,l i ,l i ) Among them, z i Used to represent the parts of the image visual features that are related to different category features.

6. The image classification management method based on semi-supervised scene recognition according to claim 1 is characterized in that: The implementation process of step (5) is as follows: In a limited set of labeled image samples, the co-occurrence relationship between different categories can assist model prediction. By calculating the number of co-occurrences of different categories, a co-occurrence matrix S of size C×C can be obtained: Among them, C represents the number of categories, e kl Indicates the number of samples of the kth category and the lth category that appear simultaneously, n k represents the number of samples that only contain the kth category; cluster analysis is performed on the co-occurrence matrix S, and the original label space is divided into several clusters according to the co-occurrence of different categories, where each cluster can represent a separate scene; specifically, the co-occurrence matrix is ​​first used to obtain the similarity matrix P of the category co-occurrence relationship diagram: After the scene is divided, it is expressed as a standardized graph cutting problem: Where D is a diagonal matrix, and its diagonal elements can be expressed as At the same time, L = DP is the Laplacian matrix; F represents the learned embedding representations of different categories, which contains the co-occurrence information of the categories. It represents the k smallest eigenvalues; when the trace of the matrix is ​​the smallest, Using the k-means clustering operation, the original label space can be divided into K subsets. For each labeled sample, the scene c to which it belongs i The clustering results can be used to make a judgment.

7. The image classification management method based on semi-supervised scene recognition according to claim 1 is characterized in that: The implementation process of step (6) is as follows: Using the global features of the image and the label grouping results, the loss function of the scene identification auxiliary task is obtained according to the paradigm of semi-supervised learning: Among them, τ is the threshold set for the scene identification auxiliary task, L ce is the cross entropy loss; only when the predicted probability of the output is greater than the threshold will the model take the loss of the sample into account for optimization.

8. The method for image classification management based on semi-supervised scene recognition according to claim 1, characterized in that: The implementation process of step (7) is as follows: Using category-related features z i , and after the text encoder E T (·) The extracted features of the learnable prompt words t t , calculate the dot product of the two to get the predicted probability value of the sample in different categories: Among them, σ(·) is the sigmoid activation function, which is used here and To represent the category prediction probability of weakly enhanced and strongly enhanced versions of samples respectively; then use a separate threshold for each category to pseudo-label the unlabeled samples The specific calculation process is: in, and They are used to represent the thresholds of positive and negative samples in unlabeled samples, q jk Represents the predicted probability of the weakly enhanced sample on the kth category; in obtaining the pseudo label of the unlabeled data After that, the unsupervised loss of n unlabeled samples belonging to C categories is calculated as: Predict the probability p based on the picture k and label y k , define L(p k ,y k )=y k L + (p k )+(1-y k )L-(p k ) is the multi-label loss, γ1 and γ2 are used to adjust the degree of attention paid by the loss function to positive and negative samples; the supervised loss of m labeled samples belonging to C categories can be expressed as:

9. The method for image classification management based on semi-supervised scene recognition according to claim 1, characterized in that: The total loss function of step (8) is: L=L aux +L sup +L unsup Among them, L aux is the scene identification auxiliary task loss, L sup For the supervised loss of labeled samples, L unsup is the unsupervised loss for unlabeled samples.

Images