Picture classification management method based on semi-supervised scene recognition

Abstract

The application discloses a picture classification management method based on semi-supervised scene recognition, introduces a visual-language cross-modal model, defines a labeled picture set and an unlabeled picture set as a model training set, extracts picture visual features and picture text label features, extracts an embedding after picture visual feature map pooling as global information of a picture sample, inputs the embedding into a scene discriminator to obtain a scene discrimination result, inputs an embedding before picture visual feature map pooling and calculates category related features, divides a label space into several clusters by using a co-occurrence relationship of categories in a picture data set, simultaneously introduces an auxiliary task of scene recognition, divides categories into different clusters by using co-occurrence information in labels, inputs picture global features to discriminate scenes, and obtains a loss function. The application can improve the speed and precision of picture classification, simultaneously provides a scene keyword search function, and helps users to quickly search and position specified pictures.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of picture classification, and particularly relates to a picture classification management method based on semi-supervised scene recognition. BACKGROUND

[0002] The development of deep learning has led to the wide application of deep learning networks in different industries, such as recommendation systems, medical diagnosis, and video and picture labeling practical application scenarios. However, the condition necessary for training a deep learning model is a large amount of accurately labeled data. However, in real-world scenarios, obtaining a large amount of accurately labeled data requires a large amount of resources. In the scenario of picture classification management, if a traditional scheme is used to realize accurate classification and grouping of pictures, a data engine needs to be constructed to complete batch labeling of data. This operation includes multiple complex operations such as data collection, data cleaning, feature extraction, manual or machine-assisted labeling, and labeling quality auditing. The process is complex and requires high funding support. In addition, in user album management, due to privacy issues, it is not practical to conduct large-scale data labeling. Therefore, in the real picture classification management scenario, data is generally composed of a small number of accurately labeled samples and a large number of unlabeled samples, which means that limited supervised information needs to be used to train the model to realize accurate and rapid classification of pictures.

[0003] Therefore, in order to cope with these challenges, the application provides a picture classification management method based on semi-supervised scene recognition, which can efficiently utilize the limited labeled picture supervision information provided by the user, use the semi-supervised learning paradigm to help the model learn on unlabeled picture samples, and realize classification and management of pictures. SUMMARY

[0004] The application aims to realize classification of a large number of pictures and further grouping according to scene information to realize rapid retrieval. The application provides a picture classification management method based on semi-supervised scene recognition.

[0005] Technical scheme: The picture classification management method based on semi-supervised scene recognition provided by the application comprises the following steps:

[0006] (1) Given a picture dataset D, containing a labeled picture set D l provided by the user and an unlabeled picture set D u , as a visual-linguistic cross-modal model training set, and initializing the visual-linguistic cross-modal model optimizable parameters;

[0007] (2) Randomly selecting B samples from the training set, including labeled sample pictures and unlabeled sample pictures; simultaneously using the visual encoder E V (·) and the text encoder E T(·) extract picture visual features and picture text label features respectively;

[0008] (3) extract the embedding after the pooling of the visual feature map as the global information of the picture sample, and input it into the scene discriminator to obtain the scene discrimination result;

[0009] (4) input the embedding before the pooling of the visual feature map and calculate the category-related feature;

[0010] (5) divide the label space into several clusters by using the co-occurrence relationship of the categories in the picture data set, and each cluster represents the scene information;

[0011] (6) calculate the auxiliary loss L aux in combination with the global picture feature obtained in step (3) and the scene division result in step (5);

[0012] (7) use the category-related feature extracted in step (4) to calculate the supervised loss L sup and the unsupervised loss L unsup according to whether the sample has a label;

[0013] (8) optimize the total loss function and minimize it as the training target, update the optimizable prompt word parameters, and return to step (2) for iterative optimization, so that the visual-linguistic model finally reaches a convergent state.

[0014] Further, the step (1) is implemented as follows:

[0015] There is a labeled picture data set with a size of m and an unlabeled picture data set with a size of n , wherein y i is the label of the picture sample, and one or more elements in the label vector are set to 1, indicating that a single picture contains one or more objects;

[0016] Initialize the visual-linguistic cross-modal model optimizable parameters: initialize the model classifier by constructing an optimizable prompt word t composed of category names, and input the optimizable prompt word into the text encoder E T (·) of the visual-linguistic model to extract the text feature.

[0017] Further, the step (2) is implemented as follows:

[0018] Input the optimizable prompt word t with the category name into the text encoder E T (·) to extract the text feature w, and input the picture into the visual encoder E V (·) to extract the visual feature f i ; the two features can be represented as:

[0019] f i =E V (x i )

[0020] w = E T (t).

[0021] Furthermore, the implementation process of step (3) is as follows:

[0022] The visual branch of the visual-language model contains an attention pooling layer, located in the visual encoder E. V After (·), visual feature f i The input to the AttenPool layer can simultaneously extract global features g. i and local features l i :

[0023] [g i ,l i ] = AttenPool(f i )

[0024] global feature g i The input is fed into the scene discriminator to obtain the model's predicted probability for the current scene; for samples that have undergone strong augmentation, the discriminator outputs a predicted probability of p. a The predicted probability output by the weakly enhanced sample discriminator is q. a The scene discriminator's classification result for the scene is then expressed as: This indicates the scene group to which the image belongs, allowing users to quickly search for images.

[0025] Furthermore, the implementation process of step (4) is as follows:

[0026] Using local image features extracted from visual branches i And an optimizable cue word t composed of category names, using a cross-attention mechanism to calculate the category information contained in the local feature map of the image; using t as the query vector, l i As key vectors and value vectors, the specific calculation process is as follows:

[0027] z i =TransFormerDec(t,l i ,l i )

[0028] Among them, z i Used to represent the parts of an image's visual features that are related to different categories of features.

[0029] Further, the step (5) is implemented as follows:

[0030] In the limited labeled picture sample set, the co-occurrence relationship between different categories can assist the model prediction, and the co-occurrence matrix S of size C×C can be obtained by calculating the co-occurrence times of different categories:

[0031]

[0032] Where C represents the number of categories, e kl represents the number of samples that appear simultaneously in the kth category and the lth category, n k represents the number of samples that contain only the kth category; the co-occurrence matrix S is subjected to cluster analysis, and the original label space is divided into several clusters according to the co-occurrence of different categories, wherein each cluster represents a separate scene; specifically, the co-occurrence matrix is used to obtain the similarity matrix P of the category co-occurrence relationship graph:

[0033]

[0034] After the scene division, it is represented as a standard graph cutting problem:

[0035]

[0036] Where D is a diagonal matrix, and the diagonal elements can be represented as L=D-P is a Laplacian matrix; F represents the learned embedding representation of different categories, which contains the co-occurrence information of the categories, represents the k smallest eigenvalues; when the trace of the matrix in the formula is minimum, the Using the k-means clustering operation, the original label space can be divided into K subsets, and for each labeled sample, the scene c i can be determined by the clustering result.

[0037] Further, the step (6) is implemented as follows:

[0038] Using the picture global features and the label grouping result, the loss function of the scene discrimination auxiliary task is obtained according to the semi-supervised learning paradigm as:

[0039]

[0040] Where τ is the threshold value set for the scene discrimination auxiliary task, L ce is the cross-entropy loss; only when the output prediction probability is greater than the threshold value, the model will count the loss of the sample to optimize.

[0041] Further, the step (7) is implemented as follows:

[0042] with category-related features z i and text encoder E T (·) the features t of the extracted learnable prompt t , the dot product of the two is calculated, that is, the prediction probability value of the sample in different categories:

[0043]

[0044] where σ(·) is the sigmoid activation function, and and are used to represent the category prediction probability of the weak and strong enhanced version samples respectively; then use the threshold of each category to pseudo-label the unlabeled samples The specific calculation process is:

[0045]

[0046] where and are used to represent the threshold of positive and negative samples in the unlabeled sample, respectively, and q jk represents the prediction probability of the weak enhanced sample in the kth category; after obtaining the pseudo-label of the unlabeled data Then, the unsupervised loss of the n unlabeled samples belonging to C categories is:

[0047]

[0048] According to the picture prediction probability p k and the label y k , define L(p k , y k ) = y k L + (p k ) + (1-y k ) L - (p k ) as the multi-label loss, γ1 and γ2 are used to adjust the attention of the loss function to positive and negative samples; then the supervised loss of the m labeled samples belonging to C categories can be represented as:

[0049]

[0050] Further, the total loss function of step (8) is:

[0051] L = L aux + L sup + L unsup

[0052] wherein, L aux is the scene discrimination auxiliary task loss, L sup is the supervised loss for the labeled samples, L unsup is the unsupervised loss for the unlabeled samples.

[0053] Advantages: Compared with the prior art, the advantages of the present application: the present application can realize fine labeling of pictures and improve the accuracy of picture classification by using the rich knowledge obtained by the visual-linguistic model in the pre-training stage; during the training process, the model visual and language branch model parameters are kept frozen, only the initializable prompt words are trained, which greatly reduces the training overhead, and by using the knowledge implied in the visual-linguistic model, the labeling cost can be significantly reduced, the construction of a complex data engine is saved, and only limited supervised information is used to complete model training, so the present application has excellent application value and prospect in real life scenarios where computing resources are severely limited, such as mobile phones, tablet computers, and other devices, which can be used for automatic grouping and searching of photos in the photo album, and can be widely used in different scenarios; at the same time, based on the multi-modal design, the present application can give the text label corresponding to the picture to assist the subsequent picture grouping task according to the scene, and provide users with convenient and fast retrieval function; in addition, the present application introduces the auxiliary task of scene discrimination, and cross-attention operation between different modal features to further improve the alignment degree between modalities, reduce the dependence on a large amount of accurate labeling data, meet the needs of actual application scenarios, so that the model can complete the classification and scene grouping management of pictures only by using limited user labeling information, and improve the accuracy and speed of picture classification. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 is a flowchart of the present application. DETAILED DESCRIPTION

[0055] The present application will be further described in detail below in conjunction with the accompanying drawings.

[0056] As Figure 1 shown, the present application proposes a picture classification management method based on semi-supervised scene recognition, which fine-tunes a pre-trained visual-linguistic model on a large number of samples to realize application in the field of picture management, specifically including the following steps:

[0057] Step 1, given the labeled set D l and the unlabeled set D u as the model training set, and initialize the model optimizable parameters.

[0058] Specifically, the training data contains a labeled picture data set and an unlabeled picture data set where y is the label for multi-label samples, and there are multiple elements in the vector set to 1, indicating the semantic information of multiple objects contained in a single picture sample. The initialization model component includes: an optimizable prompt word and a scene discriminator. The model classifier is initialized by constructing an optimizable prompt word t composed of category names, and the optimizable prompt word is input into the text encoder E i T (·) extracts text features as the final category classifier. The input picture is input into the visual encoder E V (·) in the model to extract features, and then the category classifier is used to obtain the prediction probability of the model in different categories.

[0059] In this embodiment, image management in a mobile phone album is taken as an example. 20,000 images are initially given, including a small part of a user-labeled labeled data set which contains 100 pictures with detailed label information; and a large part of an unlabeled data set The number of these unlabeled pictures reaches 19,000. These pictures cover various scenes in daily life, such as family, transportation, scenery, and several themes. In this scenario, the goal of image management is to automatically extract and analyze the semantic information of a large amount of unsupervised data through training of a small amount of labeled data, to attribute similar pictures to different groups, so that users can easily retrieve and browse image content of different categories. Specifically, each picture often contains multiple object or scene information. For example, a photo may contain both the object class label "car" and the scene class label "city street" or "highway". In addition to containing different object label information, each picture also contains scene label information, so the label vector y i of the sample has multiple elements set to 1, corresponding to the annotation of different category information.

[0060] Step 2: Randomly select B samples from the training data set, including labeled samples and unlabeled samples. At the same time, use the visual encoder E V (·) and the text encoder E T (·) in the visual-linguistic model to extract sample features f i and category prompt word features w, respectively. The two features can be represented as:

[0061] f i =E V (x i ), w=E T (t)

[0062] ​By constructing the optimizable prompt words containing the category names and inputting them into the text encoder to extract the text features as the final category classifier, the method can effectively provide prior knowledge for model optimization. In this embodiment, when predicting the category to which the object in the picture belongs, the text encoder can receive any text description as the category prior knowledge, so it can cover a wide label space given by the user in the actual scenario, such as the user wants to search for the object “bus” in the album, and the text encoder can also receive the keyword “bus”. In addition, when constructing the optimizable prompt words, the model labels can also be given as abstract concepts such as joy and sadness. Such design can serve the needs of users for abstract semantics in the actual scenario.

[0063] Step 3, extract the embedding after the pooling of the visual feature map as the comprehensive understanding of the picture sample, and input it into the scene discriminator to obtain the scene discrimination result.

[0064] The scene discrimination information provided by the user can also assist in labeling the object on the detailed object. Assuming that the provided scene information is a living room, the appearance probability of categories such as TV, remote control, dining table, and flowerpot will increase in this scene. Therefore, the scene information in the labeling information can be used to better serve the grouping of the album pictures. There is an attention pooling layer in the visual branch of the visual-linguistic model, which is located after the visual encoder E V (·) and before the visual feature f i is input into the attention pooling layer, the global feature g i and the local feature l i can be extracted at the same time:

[0065] [g i , l i ]=AttenPool(f i )

[0066] The global feature g i is input into the scene discriminator to obtain the prediction probability of the model for the current scene, which represents the overall understanding of the model for the picture. If the input is a no-label sample, the prediction probability output by the discriminator for the strongly enhanced sample is p a , and the prediction probability output by the discriminator for the weakly enhanced sample is q a . The classification result of the scene discriminator for the scene can be represented as: Finally can be used to represent the scene to which the no-label sample belongs.

[0067] Step 4, input the embedding before the pooling of the visual feature map and calculate the category-related features.

[0068] In actual album management scenarios, the mechanism for calculating category-related features plays a significant role in large-scale automatic sorting and classification of pictures. When a user uploads a large number of pictures, the system can group and label the pictures in detail by automatically extracting local features from each picture in combination with category prompt words (such as "wedding", "travel", "pet", "landscape", etc.). For example, for a picture containing multiple scenes, the system can accurately identify multiple category features such as "crowd", "beach", and "umbrella" through a cross-attention mechanism, and generate multi-label annotations containing these key information. This not only improves the accuracy of picture grouping, but also provides more dimensional detail information in labeling, greatly reducing the burden of manual sorting by the user.

[0069] Specifically, the present application uses picture local features l i extracted from the visual branch i and an optimizable prompt word t composed of category names to calculate the category information contained in the picture local feature map using a cross-attention mechanism. Take t as the query vector, l i as the key vector and the value vector, and the specific calculation process is as follows:

[0070] z i = TransFormerDec(t, l i )

[0071] z i , which can be used to represent the part of the picture visual features related to different category features. In the multi-label learning scenario, this method can further improve the model's ability to identify category features and improve the alignment between visual modalities and text to ensure that the picture and text description in the album application are more consistent and coherent. This is manifested in actual operation as the system can intelligently generate picture labels or annotations through text prompts to ensure that the description is closely related to the image content. For example, for a picture taken at a dinner party, the system can not only detect "dinner table", "food", "candlelight", and other visual elements in the picture, but also generate highly semantic labels such as "warm dinner" or "family reunion". This allows users to quickly locate relevant pictures through keywords or categories when searching and browsing the album, greatly improving management efficiency.

[0072] Step 5: Divide the label space into several clusters using the co-occurrence relationship of categories in the data set, and each cluster can represent scene information.

[0073] In the context of album management, the co-occurrence relationship between different categories can effectively help the model make more accurate category predictions. In daily life photos, many categories often appear simultaneously, for example, in photos of family gatherings, "table", "food" and "person" often appear together. This co-occurrence pattern between categories can provide additional information for the album system to better predict and organize images. The co-occurrence relationship between different categories can assist the model prediction, and by calculating the co-occurrence times of different categories, a co-occurrence matrix S of size CxC can be obtained:

[0074]

[0075] where C represents the number of categories, here e kl represents the number of samples that appear simultaneously in the kth category and the lth category, n k represents the number of samples containing only the kth category. After clustering analysis of the co-occurrence matrix S, the original label space can be divided into several clusters according to the co-occurrence of different categories, where each cluster can represent a separate scene. For example, pictures containing "beach", "sea waves" and "palm trees" can be divided into "seaside vacation" scenes, while pictures containing "person", "cake" and "candle" can be divided into "birthday party" scenes. In this way, the system can compress and organize the original label space according to the co-occurrence of categories in the pictures, thereby improving the efficiency of picture management and classification. Specifically, first, the co-occurrence matrix is used to obtain the similarity matrix P of the category co-occurrence relationship graph:

[0076]

[0077] The scene division can then be represented as a standard graph cut problem:

[0078]

[0079] where D is a diagonal matrix, and its diagonal elements can be represented as L=D-P is the Laplacian matrix. F represents the learned embedding representation of different categories, which contains the co-occurrence information of the categories, represents the k smallest eigenvalues. By using the k-means clustering operation, the original label space can be divided into K subsets, and for each labeled sample, its belonging scene c i can be determined by the clustering result.

[0080] Step 6, calculate the auxiliary loss L aux based on the global image features obtained in step 3 and the scene division result in step 5.

[0081] By leveraging global image features and label grouping results obtained from category co-occurrence analysis, the system can further design scene identification auxiliary tasks, thereby improving its ability to recognize image categories and scenes. By integrating this task into a semi-supervised learning framework, it can effectively utilize large amounts of unlabeled data, improving the model's generalization ability and classification performance. Assuming the labeled set has m samples and the unlabeled set has n samples, the loss function is:

[0082]

[0083] Where τ is the threshold set for the scene identification assistance task; the model will only include the loss of that sample in its optimization if the output predicted probability is greater than this threshold. ce Cross-entropy loss measures the difference between the probability distribution predicted by the model and the real-world scene labels. By minimizing this loss, the model can gradually learn how to utilize global features and label grouping results to accurately identify different scenes. This provides a more intelligent and refined method for image organization and classification in photo album management systems.

[0084] Step 7: Using the category-related features extracted in Step 4, calculate the supervised loss L based on whether the sample has a label. sup And unsupervised loss L unsup .

[0085] Using the obtained class-related features z i And after passing through the text encoder E T The features of the learnable cue words extracted by (·) t Calculating the dot product of the two yields the predicted probability values ​​of the sample in different categories.

[0086]

[0087] Where σ(·) is the sigmoid activation function, used here... and Let represent the predicted class probabilities of the weakly and strongly enhanced versions of the sample, respectively. Then, a separate threshold for each class is used to assign pseudo-labels to the unlabeled samples. The specific calculation process is as follows:

[0088]

[0089] in and q are used to represent the thresholds for positive and negative samples in the unlabeled sample set, respectively. jk This represents the predicted value for the k-th category. The pseudo-label is obtained from the unlabeled data. Then, the unsupervised loss can be calculated as follows:

[0090]

[0091] According to the picture prediction probability p k and the real label y k , define L(p k , y k ) = y k L + (p k ) + (1-y k )L - (p k ) as the multi-label loss, where γ1 and γ2 can be used to adjust the degree of attention of the loss function to positive and negative samples. The supervised loss can also be expressed as:

[0092]

[0093] Step 8: Update the optimizable prompt word parameters and return to step 2 for iterative optimization, and finally make the model converge.

[0094] L = L aux + L sup + L unsup

[0095] Where L aux is the scene discrimination auxiliary task loss, L sup is the supervised loss for labeled samples, and L unsup is the unsupervised loss for unlabeled samples. According to the above steps, the model can complete the first round of optimization. Then the model can follow the same steps to iteratively optimize until convergence, and finally the optimized prompt words can be used as the final classifier of the model. In actual application scenarios, since the parameters to be optimized only exist in a small part of the prompt words, the high computing power requirement and time overhead are saved, and after a new picture appears in the album, the user can quickly get the class label of the new picture and the scene group to which it belongs.

[0096] The present application is particularly suitable for large-scale picture management systems, especially in the context of photo album applications on mobile devices. As the number of pictures stored in a user's mobile phone album increases rapidly, it becomes extremely difficult to manually classify and manage the pictures. Through the intelligent picture classification method based on category co-occurrence analysis and semi-supervised learning proposed by the present application, users can automatically and efficiently group a large number of pictures by scene and category, reducing the tedious operation of manual sorting. The method of extracting category-related features in the sample proposed in the present application can enhance the interaction between different modalities, and the auxiliary task of scene discrimination can greatly help users in the actual need of managing pictures in the photo album.

[0097] To verify the superiority of the present method, the model is evaluated on two commonly used multi-label picture datasets: MS-COCO 2014 (COCO) and VOC-2012 (VOC). These two datasets cover a large number of pictures commonly seen in daily life, simulating the user's need for fine annotation and scene grouping of a large number of pictures. Existing semi-supervised multi-label methods are collected for comparison. Among the comparison methods, BCE and ASL are commonly used loss functions in multi-label learning; DualCoOp, SCPNet, TaI and DualCoOp++ are also methods based on fine-tuning visual-linguistic models.

[0098] COCO contains a total of 82081 training set pictures and 40137 test set pictures, distributed in 80 different categories. VOC contains 5717 training pictures and 5823 test pictures, containing 20 categories. In order to simulate the real album management scenario where there are different proportions of labeled and unlabeled data, the labeled data proportion p is set to {0.05, 0.1, 0.15, 0.20}. The evaluation index is the mean average precision (mAP) on the test set.

[0099] ResNet-50 is used as the image encoder backbone network of the model, and the Transformer of the visual-linguistic model is used as the text encoder. The parameters of the two backbone networks remain frozen throughout the training process. A class-specific prompt setting is adopted, that is, each class has an independent set of parameters. Two independent prompt vectors are optimized, and the length of the prompt vector that can be optimized is set to 16. In order to generate features specific to a class, a Transformer decoder consisting of 2 layers and 4 attention heads is used. The original label space is divided into K=6 clusters (for COCO) and K=2 clusters (for VOC) using the spectral clustering method. The AdamW optimizer is used as the optimizer, and the learning rate is set to 0.001. The threshold for scene discrimination is set to 0.9. The experimental results are shown in Table 1:

[0100] Table 1 mAP of different methods on different datasets

[0101]

[0102] The present application achieves the best performance in all experimental settings, especially significantly outperforms other methods in the case of small label proportion, and does not bring excessive computational cost. The performance improvement is mainly due to the introduction of the scene discrimination auxiliary task and the better feature representation obtained by aligning the target task with semantic perception to extract class-related features in the picture. The present application is suitable for a variety of practical application scenarios without introducing excessive overhead, especially in large-scale image and video management systems, users can automatically classify and label a large number of photos, and users can quickly find photos of specific scenes such as family gatherings, travel and scenery according to text labels.

[0103] The above only describes the preferred embodiments of the present application, and it should be pointed out that for ordinary skilled persons in the art, several improvements and refinements can be made without departing from the principles of the present application, and these improvements and refinements should also be considered within the protection scope of the present application.

Claims

1. A picture classification management method based on semi-supervised scene recognition, characterized in that, Comprise the following steps: (1) Given a picture dataset D, containing a set of user-provided labeled pictures D l and a set of unlabeled pictures D u , as the visual-linguistic cross-modal model training set, and initialize the visual-linguistic cross-modal model optimizable parameters; (2) Randomly pick B samples from the training set, including labeled sample pictures and unlabeled sample pictures; at the same time, use the visual encoder E V (·) and the text encoder E T (·) to extract the visual features of the pictures and the features of the picture text labels, respectively; (3) Extract the embedding after the visual feature map pooling as the global information of the picture sample, input it into the scene discriminator to obtain the scene discrimination result; (4) Input the embedding before the visual feature map pooling and calculate the category related feature; (5) Divide the label space into several clusters by using the co-occurrence relationship of categories in the picture data set, and each cluster represents the scene information; (6) Calculate auxiliary loss L combining the global picture features obtained in step (3) and the scene division result in step (5) aux ; (7) Using the class-related features extracted in step (4), the supervised loss L and the unsupervised loss L are calculated according to whether the sample has a label sup and does not have a label unsup respectively. (8) Update the optimizable prompt word parameters and return step (2) for iterative optimization, and finally make the visual- language model reach the convergence state; The step (3) is implemented as follows: There is an attention pooling layer in the visual branch of the vision-language model, which is located at the position of the visual encoder E V After (·), the visual features f i are input into the attention pooling layer AttenPool, from which global features g i and local features l i can be extracted at the same time. [g i ,l i ]=AttenPool(f i ) The global feature g i is input into the scene discriminator to obtain the predicted probability of the model for the current scene; for the strongly enhanced sample, the predicted probability output by the discriminator is p a , and the predicted probability output by the discriminator for the weakly enhanced sample is q a ; the classification result of the scene discriminator for the scene is represented as: , which represents the scene group to which the picture belongs, for the user to quickly search the picture. The step (4) is implemented as follows: Using the picture local features l extracted from the visual branch i And the optimizable prompt word t composed of category names, using the cross attention mechanism to calculate the category information contained in the picture local feature map; taking t as the query vector, l i As the key vector and the value vector, the specific calculation process is: z i = TransFormerDec(t, l i ,l i ) wherein z i for indicating the part of the visual features of the picture that is related to different category features.

2. The picture classification management method based on semi-supervised scene recognition according to claim 1, characterized in that, The step (1) is implemented as follows: a labeled picture dataset of size m and an unlabeled picture dataset of size n where y i is the label of the picture sample, with multiple elements in the label vector being set to 1, indicating multiple objects in a single picture; Initializing the visual-linguistic cross-modal model can optimize parameters: initializing the model classifier by constructing an optimizable prompt word t consisting of a class name, inputting the optimizable prompt word into the text encoder E in the visual-linguistic model T (·) Extracting text features as the final class classifier. 3.The picture classification management method based on semi-supervised scene recognition according to claim 1, characterized in that, The step (2) is implemented as follows: The optimizable prompt word t with the category name is input to a text encoder E T The text feature w is extracted in (·), and the picture is input to a visual encoder E V The visual feature f is extracted in (·) i Both features can be represented as: f i = E V (x i ) w = E T (t).

4. The picture classification management method based on semi-supervised scene recognition according to claim 1, characterized in that, The step (5) is implemented as follows: In the limited labeled picture sample set, the co-occurrence relationship between different categories can assist the model prediction, and the co-occurrence matrix S of size C×C can be obtained by calculating the co-occurrence times of different categories: where C denotes the number of categories, e kl represents the number of samples that both the kth category and the lth category appear simultaneously, n k represents the number of samples that only contain the kth category; the co-occurrence matrix S is subjected to cluster analysis, and the original label space is divided into several clusters according to the co-occurrence of different categories, wherein each cluster can represent a separate scene; specifically, first, the co-occurrence matrix is used to obtain a similarity matrix P of the category co-occurrence relationship graph: After scene division, it is represented as a standard graph cut problem: where D is a diagonal matrix, whose diagonal elements can be represented as Meanwhile, L = D - P is the Laplacian matrix; F represents the learned embedding representation of different categories, which contains the co-occurrence information of categories, represents the k' smallest eigenvalues; when the trace of the matrix in the formula is minimum, the Using the operation of k-means clustering, the original label space can be divided into K subsets, and for each labeled sample, its belonging scene c i can be judged through the clustering result.

5. The picture classification management method based on semi-supervised scene recognition according to claim 1, characterized in that, The step (6) is implemented as follows: According to the semi-supervised learning paradigm, the loss function of the scene discrimination auxiliary task is obtained as follows: where τ is a threshold set for the scene discrimination auxiliary task, L ce is the cross-entropy loss; only when the predicted probability of the output is greater than the threshold, the model will count the loss of the sample to optimize.

6. The picture classification management method based on semi-supervised scene recognition according to claim 1, characterized in that, The step (7) is implemented as follows: Utilizing class-related features z i and passing through a text encoder E T (·) the extracted features t of the learnable prompt t , the dot product of the two is calculated, that is, the predicted probability value of the sample in different categories: where σ(·) is the sigmoid activation function, and we use and to denote the class prediction probabilities of weakly and strongly augmented versions of the sample, respectively; we then use a separate threshold for each class to pseudo-label the unlabeled samples The detailed computation is as follows: where, and respectively represent the threshold values for positive and negative samples in the unlabeled samples, q jk represents the predicted probability of the weakly augmented sample on the k-th class; after obtaining the pseudo labels of the unlabeled data After that, the unsupervised loss of the n unlabeled samples respectively belonging to C classes is calculated as: According to the picture prediction probability p k and the label y k , define L(p k , y k ) = y k L + (p k ) + (1-y k ) L - (p k ) as a multi-label loss, γ1 and γ2 are used to adjust the attention degree of the loss function to positive and negative samples; then the supervised loss of m labeled samples respectively belonging to C categories can be expressed as:

7. The picture classification management method based on semi-supervised scene recognition according to claim 1, characterized in that, The total loss function of the step (8) is: L = L aux + L sup + L unsup wherein L aux is the scene discrimination auxiliary task loss, L sup is the supervised loss for labeled samples, L unsup is the unsupervised loss for unlabeled samples.

Images