Small sample multi-element scene image recognition method based on pre-training cross-modal retrieval

Abstract

The application discloses a small sample multi-element scene image recognition method based on pre-training cross-modal retrieval, and steps of the method comprise: 1) establishing an image database, including images of multiple general scene categories and images of several target scene categories, each scene category comprising multiple images, and each image being provided with a corresponding scene category label; 2) extracting features of each scene category image in the image database by using a scene recognition model to obtain a feature database corresponding to each scene category; and generating a feature vector according to features of the same image in the feature database; 3) for an image A to be classified, extracting features of the image A by using the scene recognition model and generating a feature vector; and 4) the scene recognition model performs similarity calculation on the feature vector of the image A and feature vectors in each of the feature databases respectively; and taking a scene category of an image corresponding to a feature vector with the highest similarity as a scene category of the image A.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology and relates to a scene recognition method, particularly a small-sample multi-element scene image recognition method based on pre-trained cross-modal retrieval. Background Technology

[0002] Scene recognition classifies images into predefined scene categories. It often categorizes images based on their environmental content, objects, and their layout, such as kitchen, forest, and living room. It helps machines understand the surrounding environment of an image, aiding in their understanding of past and future events. It has wide applications in various technological fields, such as intelligent robots, autonomous driving, and intelligent video surveillance. Existing scene recognition technologies often employ end-to-end deep learning frameworks, directly outputting predicted probability values ​​for various scenes within an image. Related improvements are typically made to optimize image features.

[0003] Methods based on global CNN features predict the scene category directly from the entire scene image. Typically, global CNN features are extracted from input images by a CNN model. This model is first pre-trained on a large-scale dataset (see references: Deng J, Dong W, Socher R, et al. Imagenet: A large-scale hierarchical image database [C] / / 2009 IEEE conference on computer vision and pattern recognition.leee, 2009: 248-255; Zhou B, Lapedriza A, Xiao J, et al. Learning deep features for scene recognition using places database [J]. Advances in neural information processing systems, 2014, 27: 1-9; Zhou B, Lapedriza A, Khosla A, et al. Places: A 10 million image database for scene recognition [J]. IEEE transactions on pattern analysis and machine intelligence, 2017, 40(6): 1452-1464), and then fine-tuned on the target dataset (see references: Xiao J, Hays J, Ehinger KA, et al.). al. Sundatabase: Large-scale scene recognition from abbey to zoo [C] / / 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE, 2010: 3485-3492; Quattoni A, Torralba A. Recognizing indoor scenes [C] / / 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2009: 413-420). As deep networks become deeper and more complex, this has enabled global representations from these networks to achieve better performance on many tasks, including scene classification.In addition to general-purpose CNNs, some scene-specific CNNs are designed to extract global features. Zuo et al. proposed hierarchical LSTM (HLSTM) to describe contextual relationships (see Zuo Z, Shuai B, Wang G, et al. Learning contextual dependence with convolutional hierarchical recurrent neural networks[J]. IEEE Transactions on Image Processing, 2016, 25(7):2983-2996). Liu et al. proposed the DL-CNN model to extract sparse global features from the entire scene image (see Xie L, Lee F, Liu L, et al. Scene recognition: A comprehensive survey[J]. Pattern Recognition, 2020, 102:107205). Hayat et al. designed a spatial unstructured layer to address the challenges of large-scale spatial layout deformation and scale variation (refer to Hayat M, Khan SH, Bennamoun M, et al. A spatial layout and scale invariant feature representation for indoor scene classification[J].IEEE Transactions on Image Processing,2016,25(10):4829-4841).

[0004] Researchers have found that features from different layers are complementary. Low-layer features typically capture small objects, while high-layer features capture large objects (see Wu R, Wang B, Wang W, et al. Harvesting discriminative meta objects with deep CNN features for scene classification[C] / / Proceedings of the IEEE International Conference on Computer Vision.2015:1287-1295). While using all features from different layers may seem to improve the final network performance, it can also lead to overfitting and thus compromise performance. Therefore, many methods extract features only from certain layers. For example, Xie et al. constructed two dictionary-based representations to classify auxiliary scene images using CFV and MLR (see Xie GS, Zhang XY, Yan S, et al. Hybrid CNN and dictionary-based models for scene recognition and domain adaptation[J].IEEE Transactions on Circuits and Systems for Video Technology,2015,27(6):1263-1274). Tang et al. divided the GoogLeNet layers into three parts from bottom to top and extracted the final feature map of each part. Liu et al. extracted feature maps from each residual block in ResNet.

[0005] Describing complex scenes using only a single, compact feature representation is a challenging task. Researchers have found that features generated by networks trained on different datasets are often complementary. Herranz et al. discovered the optimal ratio of responses between object-centered CNNs and scene-centered CNNs. This finding has been widely applied; for example, Wang et al. used object-centered CNNs to carry information about objects depicted in an image, while scene-centered CNNs were used to capture global scene information. Following this approach, Wang et al. designed PatchNet, a weakly supervised learning method that uses image-level supervision information as a supervisory signal for effectively extracting block-level features. To improve recognition performance, Scene-PatchNet and Object-PatchNet are used together to extract features for each image patch.

[0006] Despite decades of development in scene recognition technology, most methods still fall short of the requirements for video and image scenes with rapidly changing categories and special locations (such as fixed shooting areas, different shooting angles of the same location, and unchanged layouts).

[0007] 1. Existing scene recognition models are trained using a closed label set, which means they can only recognize scenes within the label set and do not support the recognition of newly emerging scene categories.

[0008] 2. Existing scene recognition models have low interpretability. Because previous models mostly used end-to-end deep learning models, they only output the predicted probability values ​​of each scene and could not provide the basis for the model to make scene classification judgments.

[0009] 3. Existing scene recognition models mostly recognize general scenes, such as natural scenes and man-made scenes (indoor / outdoor), and do not support scene recognition near a specific location or for the target's functional intent. Summary of the Invention

[0010] To address the problems existing in the prior art, the present invention aims to provide a small-sample multi-feature scene image recognition method based on pre-trained cross-modal retrieval. To enable the model to possess strong scene recognition capabilities, the present invention employs pre-training technology to train a powerful visual feature extraction module.

[0011] The technical solution of this invention is as follows:

[0012] A few-sample multi-feature scene image recognition method based on pre-trained cross-modal retrieval, the steps of which include:

[0013] 1) Establish an image database, which includes images of multiple general scene categories and images of several target scene categories. Each scene category includes multiple images, and each image is assigned a corresponding scene category label.

[0014] 2) Use a scene recognition model to extract features from the images of each scene category in the image database to obtain a feature database corresponding to each scene category; generate a feature vector based on the features of the same image in the feature database;

[0015] 3) For an image A to be classified, the scene recognition model is used to extract the features of image A and generate a feature vector;

[0016] 4) The scene recognition model calculates the similarity between the feature vector of image A and the feature vectors in each of the feature databases; the scene category of the image corresponding to the feature vector with the highest similarity is taken as the scene category of image A.

[0017] Furthermore, for each scene category, several scene feature keywords and scene feature objects related to the characteristics of that scene category are set; the scene recognition model includes a scene text recognition model and a scene object recognition model; the scene text recognition model includes a text detection model and a character recognition model. After the text detection model detects the text in image i, it matches it with the scene feature keywords of the scene category corresponding to image i, and uses the detected matching text as the text feature of image i; after the scene object recognition model detects the candidate object in image i, it matches it with the scene feature objects of the scene category corresponding to image i, and extracts the features of the detected matching object as the object feature of image i.

[0018] Furthermore, for newly added scene categories, several images of the newly added scene category are obtained and added to the image database, and the features of the corresponding images are extracted using a scene recognition model to generate feature vectors, which are then added to the feature database of the newly added scene category.

[0019] Furthermore, the PP-LCNet model is distilled using the PP-SSLD distillation method to obtain the scene recognition model; the features extracted from the scene recognition model are processed using arcmargin to generate hash values, and then the obtained hash values ​​are quantized using PACT, outlier points are removed, and then activated to obtain the feature vector.

[0020] Furthermore, the IVF, HNSW32, or Flat algorithm is used to extract feature vectors from each of the aforementioned feature databases and perform similarity calculations with the feature vectors of image A.

[0021] Furthermore, the scene recognition model is a pre-trained visual backbone network.

[0022] Furthermore, the target scene category image is an image that has the characteristics of the target scene.

[0023] A server is characterized by comprising a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the steps of the methods described above.

[0024] A computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the steps of the above-described method.

[0025] This invention addresses the problem of closed scene label sets in previous methods by proposing an open-domain (arbitrary scene label set) scene recognition algorithm. First, a strong visual feature extraction module is obtained through pre-training with a large amount of data. Then, by transforming classification into retrieval, a vector library is established for the scene labels to be recognized, supporting image recognition of arbitrary scene labels.

[0026] This invention addresses the problem of low interpretability in scene recognition models by proposing an interpretable scene recognition algorithm. While providing scene image labels, the model also identifies elements within the regions of interest in the image, providing the specific factors that inform the model's judgment and thus enhancing its interpretability.

[0027] This invention proposes a few-sample scene recognition algorithm based on a retrieval approach, targeting the identification of unique locations and specific intent scenarios. It supports image recognition of newly labeled images with only a small number of sample images.

[0028] In the feature extraction stage, this invention uses a large-scale pre-trained visual model as the feature extractor. Compared to features obtained using descriptors and bag-of-words vectors, the features extracted by the model, which has been pre-trained on a large number of images, are more robust and discriminative.

[0029] This patent directly uses the feature vectors extracted from a large model as the retrieval object, eliminating the need for multiple clustering and feature recoding (bag-of-words model) to obtain the retrieval object, thus significantly saving time. Furthermore, this patent stores multiple feature vectors from different perspectives for each scene image category. Compared to a single feature vector, category determination can be completed simply by ensuring that the features of the test image have a high similarity to one of the multiple features of each scene category, thereby improving the accuracy and robustness of the model retrieval.

[0030] This patent employs a faster vector retrieval algorithm. Compared to traversing all feature vectors one by one, this patent uses IVF, HNSW32, and Flat algorithms, which reduces the algorithmic complexity of the retrieval.

[0031] The advantages of this invention are as follows:

[0032] 1. Compared to existing technologies that only support scene recognition with closed label sets, this proposal is based on the idea of ​​classifying into retrieval. By performing similarity retrieval between the image to be recognized and the existing labeled images in the image database, it supports scene recognition in the open domain.

[0033] 2. Compared to existing scene recognition models that often only provide probability values ​​for classification labels, this proposal also provides the criteria for judging the elements of model classification.

[0034] 3. Compared to existing models that mostly identify general scenarios, this proposal supports the identification of scenarios with specific locations and intentions.

[0035] 4. Compared to existing models that often require retraining when recognizing new scenes, this proposal only needs to store a small number of new scene sample images to complete image recognition. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of small-sample multi-factor scene image recognition based on pre-trained cross-modal retrieval.

[0037] Figure 2 This is a flowchart of a scene recognition algorithm based on pre-trained cross-modal retrieval.

[0038] Figure 3 This is a flowchart of an interpretable scene recognition algorithm based on multiple elements. Detailed Implementation

[0039] The present invention will now be described in further detail with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0040] This invention proposes a small-sample, multi-element scene image recognition method based on pre-trained cross-modal retrieval. It supports open-domain scene recognition algorithms, recognizing not only general scenes but also scenes specific to particular locations and intentions. Furthermore, it provides both the predicted probability of the image scene and the basis for the model's classification judgment. Additionally, it supports storing a small number of newly labeled sample images into the model to complete image recognition of new scenes.

[0041] The algorithm's approach is as follows: To enhance the model's scene recognition capabilities, it first pre-trains with a large amount of data to obtain a powerful visual feature extraction module. To enable the model to support open-domain scene recognition, it proposes a classification-to-retrieval approach. This involves storing images of the scene categories to be recognized, creating a vector database. When a test image is fed into the model, it is compared with the vectors in the image database, and the image with the highest similarity is selected as the predicted scene category. By storing images of specific locations and intentions in the image database, the model gains the ability to recognize specific target scenes. Simultaneously, to improve the model's interpretability, after recognizing the current image as the target scene, a scene text recognition model and a scene object recognition model are executed to identify scene features and important text in the image, providing the basis for scene classification. When the model needs to recognize a new scene category, the image database is updated with images of the new category, thus completing scene recognition with a small number of samples.

[0042] Detailed process of this invention

[0043] The process of this invention is as follows Figure 1As shown, the process includes four core steps: establishing an image database, a retrieval-based scene recognition algorithm, interpretable scene recognition based on multiple elements, and few-sample scene recognition based on database updates. These will be introduced separately below.

[0044] 1. Establishment of an image database

[0045] To support scene recognition in the open domain of the model, the core idea of ​​this invention is to transform classification into retrieval, and an essential step in this process is to establish an image database with scene labels. For each target scene to be identified, this invention collects relevant images to form an image database, using scene characteristics as the smallest unit.

[0046] To support scene recognition based on specific locations and intentions, the image database stores images of traditional general scene categories, including 365 categories such as outdoor natural scenes, outdoor man-made scenes, and indoor man-made scenes. It also includes data on eight target scene categories.

[0047] For each specific target scene category, the image database will store multiple images that have the characteristics of the target scene.

[0048] 2. Scene recognition algorithm based on pre-trained cross-modal retrieval

[0049] like Figure 2 As shown, the core idea of ​​this invention for scene recognition is based on retrieval. After establishing the image database, a pre-trained scene recognition model is used to extract features, creating feature databases for each scene category. When a query image arrives at the scene recognition model, after feature extraction, its similarity is calculated with the vectors in the feature database. The scene category with the highest similarity is selected as the prediction result, and its prediction probability value is output. The model used here is the PaddlePaddle framework.

[0050] When using a pre-trained scene recognition model for feature extraction, a series of operations are performed, including feature extraction, distillation, hash encoding, and quantization. To enhance the model's scene recognition capabilities, the public scene dataset Places (10 million images) is used for large-scale pre-training of the visual backbone network. The PP-LCNet model, a lightweight model supported on CPUs in PaddlePaddle, is used as the visual backbone network. After simple semi-supervised label distillation (SSLD), the model's accuracy is significantly improved. Simultaneously, the ArcMargin algorithm (a metric learning algorithm primarily used for model training to make the model's output features more discriminative) is used for model training, bringing feature vectors of the same scene category closer together and widening feature vectors of different categories. Then, the DeepHash algorithm is used to generate hash values ​​for each image feature. Finally, PACT (Paramometerized Clipping AcTivation) quantization is performed, minimizing the accuracy loss caused by model quantization by removing outliers before quantizing activation values. Finally, this feature vector is stored in the feature database. This invention generates a feature vector based on the features of each image for similarity calculation.

[0051] In the vector search phase, this invention provides three fast vector retrieval algorithms: IVF, HNSW32, and Flat, implemented using the spacy package. HNSW32 is a graph indexing method that divides labels into different layers during index building, resulting in high retrieval accuracy and speed. IVF, an inverted index retrieval method, primarily utilizes the inverted index concept, storing vectors under each cluster center. During each query, it finds the nearest centers, significantly improving search efficiency by reducing the search range; it is fast but slightly less accurate, and supports adding and deleting image features. FLAT is a brute-force retrieval algorithm with the highest accuracy, but its retrieval speed is slow with large datasets, and it also supports adding and deleting image features.

[0052] 3. Interpretable scene recognition based on multiple elements

[0053] like Figure 3 As shown, to improve the interpretability of scene classification, this invention performs multi-element detection after identifying the current image as the target scene. Specifically, a scene text recognition model is used to identify scene characteristic keywords in the image; simultaneously, a scene object recognition model is used to identify objects in the target scene within the image. The scene characteristic-related keywords and objects provide the basis for the model to classify the scene.

[0054] The scene text recognition model comprises two steps: text detection and text recognition. Both utilize pre-trained models from PaddlePaddle. The text detection model employs MobileNetV3, supporting lightweight CPU detection; the text recognition model uses the classic CRNN model. After recognizing the text in the target scene, keyword matching is performed against the characteristics of that scene. When relevant keywords are found, the associated text is retained and output by the model.

[0055] The scene object recognition model still adopts a retrieval-based approach, but introduces an object candidate bounding box detection process. The detection model used is PaddlePaddle's PP-Picodet, a lightweight detection model that also supports CPU detection. After detecting object candidate bounding boxes in the image, the retrieval-based detection framework is reused to extract features from the candidate bounding boxes and compare them with scene-featured objects in the image database. The label with the highest similarity score above a certain threshold is taken as the category of the candidate object.

[0056] 4. Few-shot scene recognition based on database updates

[0057] In real-world environments, numerous new scene categories emerge that the model has not encountered during training. To enable the model to recognize these new scene categories with minimal resource consumption, this proposal also supports scene recognition with limited sample sizes. In practice, we often need to identify newly emerging characteristic scenes, for which images are typically scarce. In such cases, the model needs to be able to recognize new scenes even with a limited number of images. In this proposal, simply adding new scene images to the image database and using the HNSW32 or Flat algorithm to increase the retrieval vectors enables the model to recognize these new scenes.

[0058] Algorithm Performance Demonstration

[0059] To verify the effectiveness of this invention, target scene data was constructed from the Internet. Each target scene included a training set of 100 images and a test set of 50 images. The target scenes were further divided into two types: fixed-location and variable-location. In fixed-location target scenes, the background largely remained unchanged, while the objects in the images changed; in variable-location target scenes, both the background and objects often changed.

[0060] In fixed scenarios, the model achieved a classification accuracy of 90%; in changing scenarios, the model achieved a classification accuracy of 64%. This is because in changing scenarios, the content of the image (especially the background) often changes significantly, which increases the difficulty of the model's recognition and results in a relatively low accuracy rate; while in fixed scenarios, the background of the image remains almost unchanged, thus achieving a higher accuracy rate.

[0061] In addition, for the recognition test of the new scene category, different segments of the same video were extracted as test data. After the test, the five images of the new scene category were stored in the image database, and the recognition accuracy of the same video scene was more than 90%.

[0062] Regarding scene classification time, tests conducted on a MacOS 2.3GHz dual-core Intel Core i5 CPU computer showed a single image detection time of 2.9 seconds and a batch single image detection time of 2.0 seconds. With GPU support, the optimal detection time reported in the paper is 2.0ms.

[0063] Although specific embodiments of the invention have been disclosed for illustrative purposes to aid in understanding and implementing the invention, those skilled in the art will understand that various substitutions, variations, and modifications are possible without departing from the spirit and scope of the invention and the appended claims. Therefore, the invention should not be limited to the content disclosed in the preferred embodiments, and the scope of protection claimed by the invention is defined by the claims.

Claims

1. A method for small-sample multi-feature scene image recognition based on pre-trained cross-modal retrieval, comprising the following steps: 1) Establish an image database, which includes images of multiple general scene categories and images of several target scene categories. Each scene category includes multiple images, and each image is assigned a corresponding scene category label. 2) Use a scene recognition model to extract features from the images of each scene category in the image database to obtain a feature database corresponding to each scene category; generate a feature vector based on the features of the same image in the feature database; 3) For an image A to be classified, the scene recognition model is used to extract the features of image A and generate a feature vector; 4) The scene recognition model calculates the similarity between the feature vector of image A and the feature vectors in each of the feature databases; the scene category of the image corresponding to the feature vector with the highest similarity is taken as the scene category of image A; For each scene category, several scene characteristic keywords and scene characteristic objects related to the characteristics of that scene category are defined. The scene recognition model includes a scene text recognition model and a scene object recognition model. The scene text recognition model includes a text detection model and a character recognition model. After the text detection model detects text in image i, it matches it with the scene characteristic keywords of the scene category corresponding to image i, and uses the detected matching text as the text feature of image i. After the scene object recognition model detects candidate objects in image i, it matches them with the scene characteristic objects of the scene category corresponding to image i, and extracts the features of the detected matching objects as the object features of image i. For a newly added scene category, several images of the newly added scene category are obtained and added to the image database, and the scene recognition model is used to extract the features of the corresponding images to generate feature vectors, which are then added to the feature database of the newly added scene category.

2. The method according to claim 1, characterized in that, The scene recognition model is obtained by distilling the PP-LCNet model using the PP-SSLD distillation method; the features extracted by the scene recognition model are processed using arcmargin to generate hash values, and then the hash values ​​are quantized using PACT, outliers are removed, and then activated to obtain the feature vector.

3. The method according to claim 1, characterized in that, The similarity between the feature vectors extracted from each of the aforementioned feature databases and the feature vectors of image A is calculated using the IVF, HNSW32, or Flat algorithm.

4. The method according to claim 1, characterized in that, The scene recognition model is a pre-trained visual backbone network.

5. The method according to claim 1, characterized in that, The images in the target scene category are pictures that have the characteristics of the target scene.

6. A server, characterized in that, It includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing each step of the method of any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images