Real-time feature-based image recognition method, system, device and storage medium

Abstract

The application provides a real-time feature-based image recognition method, system, device and storage medium. Sample features are extracted from input picture data through a neural network of an encoder. Recognition features with class invariance and discriminability are obtained from the sample features. Image recognition results are obtained from the recognition features. The application can update each clustering feature and class proxy feature in real time, retains the original feature distribution and original data manifold in the feature class, finally obtains recognition features with class invariance and discriminability, and greatly improves the performance of the image recognition model and the accuracy of the recognition results.

Description

Technical Field

[0001] This application belongs to the field of computer technology, and specifically relates to an image recognition method, system, device and storage medium based on real-time features. Background Technology

[0002] With the rapid development of internet technology, people have increasingly more needs for image processing, such as in security and big data analysis. In the field of image processing, nonparametric memory feature storage and contrastive learning are commonly used methods for image feature extraction and retrieval. Taking pedestrian recognition as an example, people often use surveillance cameras to record a large number of images or video clips of pedestrians to find specific pedestrians, which plays an important role in security systems and intelligent monitoring. In real-world scenarios, images taken by different camera devices have different styles, shooting angles, and pedestrian postures. Therefore, most current methods employ a momentum-updated nonparametric feature storage module to retain the feature information for each category. Then, based on pseudo-labels, the current features are aligned to the corresponding category features (positive samples) in the storage repository, while moving away from other category features (negative samples). Finally, an efficient and comprehensive contrastive learning framework is constructed to extract discriminative features from the input image, thereby better performing pedestrian image retrieval tasks.

[0003] However, current unsupervised identification methods primarily use the mean of features belonging to the same class as surrogate features for each category. Their underlying assumption is that the data for each cluster is distributed in a high-dimensional spherical distribution. If the actual data is distributed in manifolds rather than spherical clusters, using the mean feature as the surrogate feature for each category is inaccurate. Furthermore, using momentum update strategies to update feature or category surrogate points alters the original feature distribution in the data. All of these factors contribute to suboptimal performance in identification and retrieval results. Summary of the Invention

[0004] The image recognition system, method, device, and storage medium based on real-time features proposed in this invention can update each cluster feature and category proxy feature in real time, preserving the original feature distribution and original data manifold in the feature category, and finally obtaining recognition features with class invariance and discriminability, which greatly improves the performance of the image recognition model and the accuracy of the recognition results.

[0005] According to a first aspect of the embodiments of this application, an image recognition method based on real-time features is provided, comprising the following steps:

[0006] Based on the input image data, sample features are extracted through the encoder's neural network;

[0007] Based on the sample characteristics, we obtain recognition features that are class-invariant and discriminative;

[0008] Image recognition results are obtained based on the recognition features.

[0009] In some embodiments of this application, based on sample characteristics, identification features with class invariance and discriminative properties are obtained, including:

[0010] Align sample features with features of their class to learn class-invariant features; and move sample features close to surrogate features of their class to learn discriminative features.

[0011] We obtain recognition features that are class-invariant and discriminative.

[0012] In some embodiments of this application, before aligning sample features with their class features to learn class-invariant features; and before moving sample features closer to their class proxy features to learn discriminative features, the method further includes:

[0013] Store sample features as instance-level features to obtain category features; and store sample features as category-level features to obtain category proxy features.

[0014] In some embodiments of this application, sample features are extracted from the input image data using a neural network of an encoder, including:

[0015] The encoder model is pre-trained on the ImageNet dataset to obtain model weights; the encoder's neural network is then initialized based on these model weights.

[0016] The feature layers extracted from the backbone network are subjected to generalized average pooling to obtain feature vectors, which are then standardized.

[0017] Clustering algorithms are used to cluster all standardized features, and data belonging to the same cluster are assigned the same pseudo-label.

[0018] In some embodiments of this application, the sample features are aligned with the features of their respective categories, and class-invariant features are learned, including:

[0019] Given one feature from the features of a sample;

[0020] A sample-to-instance contrastive learning process is applied to the features to align them with all features of their respective categories.

[0021] In some embodiments of this application, discriminative features are learned by aligning sample features with the proxy features of their respective categories, including:

[0022] Given one feature from the features of a sample;

[0023] A contrastive learning process, from samples to categories, is applied to the features to bring them closer to the surrogate features of their respective categories.

[0024] In some embodiments of this application, sample features are stored as category-level and instance-level features to obtain the category features, including:

[0025] Construct an instance-level repository to store the features of all images in the training set; in each iteration, directly replace the feature corresponding to the instance-level repository with the current feature;

[0026] A category-level repository is constructed to store the proxy features of each category obtained by clustering algorithm in the training set. In each training cycle, the category center proxy is selected in each category for initialization. In each iteration, the category proxy features corresponding to the category-level repository are replaced by random sampling.

[0027] According to a second aspect of the embodiments of this application, an image recognition system based on real-time features is provided, comprising:

[0028] Encoder module: Used to extract sample features from the input image data through the encoder's neural network;

[0029] Recognition Feature Module: Used to obtain recognition features with class invariance and discriminative properties based on sample features;

[0030] Image recognition module: Used to obtain image recognition results based on recognition features.

[0031] According to a third aspect of the embodiments of this application, an image recognition device is provided, comprising:

[0032] Memory: used to store executable instructions; and

[0033] Processor: Used to connect to memory to execute executable instructions to complete the image recognition method.

[0034] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided having a computer program stored thereon; the computer program is executed by a processor to implement an image recognition method.

[0035] The image recognition system, method, device, and storage medium based on real-time features of this application extract sample features from input image data through a neural network of an encoder; based on the sample features, recognition features with class invariance and discriminative properties are obtained; and based on the recognition features, the image recognition result is obtained. This application can update each cluster feature and category surrogate feature in real time, preserving the original feature distribution and original data manifold in the feature categories, and finally obtaining recognition features with class invariance and discriminative properties, which greatly improves the performance of the image recognition model and the accuracy of the recognition results. Attached Figure Description

[0036] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0037] Figure 1 The diagram illustrates the steps of an image recognition method based on real-time features according to an embodiment of this application.

[0038] Figure 2 The diagram illustrates the steps for obtaining sample features and sample pseudo-labels according to an embodiment of this application.

[0039] Figure 3 The diagram illustrates the steps for obtaining identification features according to an embodiment of this application;

[0040] Figure 4 The diagram shows a flowchart of an image recognition method according to an embodiment of this application;

[0041] Figure 5 The diagram shows a pedestrian image in an image recognition scene according to an embodiment of this application;

[0042] Figure 6 The diagram shows a schematic representation of the principle of an encoder extracting sample features according to an embodiment of this application;

[0043] Figure 7 The diagram shows a flowchart of encoder feature processing according to an embodiment of this application;

[0044] Figure 8 The diagram illustrates a process for obtaining class-invariant and discriminative identification features according to an embodiment of this application;

[0045] Figure 9 The diagram shows a schematic representation of an image recognition system according to an embodiment of this application.

[0046] Figure 10 Another structural schematic diagram of an image recognition system according to an embodiment of this application is shown in the figure;

[0047] Figure 11 The diagram shows a schematic representation of an image recognition device according to an embodiment of this application. Detailed Implementation

[0048] In developing this application, the inventors discovered that current unsupervised person re-identification (Re-ID) methods primarily use the mean of features belonging to the same class as surrogate features. Their underlying assumption is that the data in each cluster is distributed in a high-dimensional spherical distribution, which is the same assumption as traditional K-means clustering. Compared to the K-means algorithm, existing methods mainly use density-based clustering (DBSCAN) to generate pseudo-labels for unlabeled data, achieving higher accuracy in Re-ID tasks. This is mainly because Re-ID data is actually distributed in a high-dimensional manifold and does not necessarily follow spherical clustering.

[0049] Based on the above analysis, the inventors found that there is a certain conflict between the clustering method and the training process in the current state-of-the-art (SOTA) method.

[0050] First, using the mean feature as a surrogate feature for each class is inaccurate. If the actual data is distributed in a manifold rather than a spherical cluster, the surrogate point for the mean feature may not lie in its own class distribution, or even in other class distributions. Second, most existing methods use momentum update strategies to update feature or class feature surrogate points, which alters the original feature distribution in the data. This assumption of a spherical distribution in the cluster, based on K-means, is also inconsistent with the DBSCAN clustering assumption used by most methods. Therefore, this leads to suboptimal performance.

[0051] Based on this, this application proposes a new real-time memory update strategy and two contrastive learning schemes for unsupervised pedestrian re-identification.

[0052] Compared with the prior art, the image recognition system, image recognition method, device and storage medium disclosed in this application based on real-time features can be applied to the recognition of pedestrian image information. During the training of the neural network, the features stored in the memory bank are directly replaced with random features sampled from the current small batch without momentum update, which enables each cluster feature to be updated in real time and preserves the original feature distribution in the category as much as possible.

[0053] Furthermore, considering that DBSCAN clustering measures instance-level similarity, the present invention proposes a sample-to-instance contrastive loss that treats any sample as an anchor point and explores all positive and negative samples stored in the instance-level memory during training. This allows our model to fully utilize global information, which helps overcome feature variations within categories and enables the model to learn to adapt to the clustering results.

[0054] In addition, this application also proposes a contrastive learning strategy from sample to category proxy, which randomly selects an image feature from the current mini-batch features as a category proxy in the non-parametric memory, and forces the current sample feature to be close to its category proxy feature and far away from other category proxy features, thus ensuring that the learned image features are discriminative.

[0055] Using randomly selected real-time features as category proxy features follows the characteristics of the DBSCAN clustering algorithm, preserving the original data manifold rather than the spherical distribution assumed in previous work.

[0056] Specifically, the image recognition system, image recognition method, device and storage medium based on real-time features of this application are used to extract sample features from the input image data through the neural network of the encoder; based on the sample features, recognition features with class invariance and discriminativeness are obtained; and based on the recognition features, the image recognition result is obtained.

[0057] In summary, this application can update each cluster feature and category proxy feature in real time, preserving the original feature distribution and original data manifold in the feature category, and finally obtaining recognition features with class invariance and discriminativeness, which greatly improves the performance of the image recognition model and the accuracy of the recognition results.

[0058] To make the technical solutions and advantages of the embodiments of this application clearer, the exemplary embodiments of this application will be described in further detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not an exhaustive list of all embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0059] Example 1

[0060] Figure 1 The diagram illustrates the steps of an image recognition method based on real-time features according to an embodiment of this application.

[0061] like Figure 1 As shown, the image recognition method based on real-time features in this application includes the following steps:

[0062] S1: Extract sample features from the input image data using the encoder's neural network; S2: Obtain class-invariant and discriminative recognition features based on the sample features; S3: Obtain the image recognition result based on the recognition features.

[0063] This application can obtain recognition features with class invariance and discriminative properties, which greatly improves the performance of image recognition models and the accuracy of recognition results.

[0064] Figure 2The diagram illustrates the steps for obtaining sample features according to an embodiment of this application.

[0065] like Figure 2 As shown, in S1, sample features are extracted from the input image data through the encoder's neural network, including:

[0066] S11: Pre-train the image data on the ImageNet dataset to obtain model weights; initialize the encoder's neural network based on the model weights.

[0067] S12: Perform generalized average pooling on the feature layers extracted from the backbone network to obtain feature vectors, and then standardize them.

[0068] S13: Use a clustering algorithm to cluster all standardized features, and assign the same pseudo-label to the data corresponding to features belonging to the same cluster.

[0069] Figure 3 The diagram illustrates the steps for obtaining identification features according to an embodiment of this application.

[0070] like Figure 3 As shown, in S2, based on the sample characteristics, the recognition features with class invariance and discriminative properties are obtained, including:

[0071] S21: Store the sample features as instance-level features to obtain the category features; and store the sample features as category-level features to obtain the category proxy features.

[0072] S22: Align sample features with their class features to learn class-invariant features; and move sample features close to class surrogate features to learn discriminative features.

[0073] S23: Obtain recognition features that are class-invariant and discriminative.

[0074] In S21, in some embodiments of this application, the sample features are stored as instance-level and category-level features to obtain the category features and category proxy features, including:

[0075] 1. Construct an instance-level repository to store the features of all images in the training set; in each iteration, directly replace the feature corresponding to the instance-level repository with the current feature.

[0076] 2. Based on the clustering algorithm, the category information of each sample feature in the training set is obtained. A category-level repository is constructed to store the surrogate features of each category obtained in the training set. In each training cycle, the category center surrogate is selected in each category for initialization. In each iteration, the category surrogate features corresponding to the category-level repository are replaced by random sampling.

[0077] In S21, the sample features are aligned with the features of their respective categories, and class-invariant features are learned, including: one feature given the sample features; and a sample-to-instance contrastive learning process is applied to the features to align the features with all features of their respective categories.

[0078] In S21, the sample features are brought closer to the surrogate features of their respective categories to learn discriminative features, including: a feature given the sample features; applying a contrastive learning process from sample to category to the feature, so that the feature is close to the surrogate features of its respective category, while moving away from the surrogate features of other categories.

[0079] To further describe the image recognition principle of this application, a specific pedestrian image recognition scenario is described in detail below.

[0080] Figure 4 The figure shows a schematic flowchart of an image recognition method according to an embodiment of this application.

[0081] like Figure 4 As shown, the model training phase mainly includes the following steps:

[0082] Step 1) Input pedestrian image data into the network model through the input module.

[0083] Step 2) Preprocessing module: Set preprocessing parameters to perform data augmentation on the input pedestrian image data.

[0084] Step 3) Encoder module: Composed of a ResNet-50 neural network, responsible for extracting feature vectors from preprocessed pedestrian images.

[0085] This application does not limit the type of neural network; other neural networks such as VGG-16 and DenseNet can be used.

[0086] Step 4) The feature storage module performs category-level and instance-level feature storage on the feature vectors extracted by the encoder module. It consists of a category-level feature storage module (Cluster-level Memory Bank, abbreviated as CMB) and an instance-level feature storage module (Instance-level Memory Bank, abbreviated as IMB).

[0087] The category-level feature storage module and the instance-level feature storage module are responsible for storing the feature vectors extracted by the encoder module at the category level and instance level, respectively.

[0088] Step 5) Sample-to-instance learning module and sample-to-category learning module.

[0089] The sample-to-instance learning module is responsible for aligning the current sample features extracted by the encoder with the category features in the instance-level feature storage module, learning class-invariant features, and mitigating intra-class feature variations for the same person. The sample-to-category learning module, on the other hand, is responsible for requiring sample features to be close to their category proxy features while moving away from other proxy category features, making the learned features more discriminative.

[0090] Secondly, the testing phase mainly includes: preprocessing the pedestrian images to be detected, inputting them into the trained model for encoding; and finally outputting the retrieval and recognition results of the pedestrian images.

[0091] Figure 5 The diagram shows a pedestrian image in an image recognition scene according to an embodiment of this application.

[0092] like Figure 5 As shown, in step 1), pedestrian images are selected from the public dataset Market1501.

[0093] The specific implementation steps for preprocessing the training dataset in step 2) are further described below.

[0094] First, resize all training images in the dataset to 256×128.

[0095] Secondly, the adjusted image was augmented in the following ways: random flipping, filling with 10 pixels, random cropping, and random erasing.

[0096] Finally, each mini-batch of image samples consists of 256 training images from 16 categories, which are sampled in a class-balanced manner and then input into the encoding module.

[0097] Figure 6 The diagram shows a schematic diagram of the principle of an encoder extracting sample features according to an embodiment of this application. Figure 7 The diagram shows a flowchart of encoder feature processing according to an embodiment of this application.

[0098] like Figure 6 As shown, the specific implementation process of the encoding module in step 3) is as follows:

[0099] First, the ResNet-50 encoding model is initialized using model weights pre-trained on the ImageNet dataset. This allows the encoding module to extract better features and accelerates the convergence of the network model.

[0100] Then, as Figure 7 As shown, the feature vectors are obtained by performing generalized average pooling on the feature layers extracted by the backbone network ResNet-50, and then standardized.

[0101] Finally, after feature extraction of the entire dataset, the DBSCAN clustering algorithm is used to cluster all features, and then the data corresponding to features belonging to the same cluster are given the same pseudo-label.

[0102] Further describing step 4), the feature vectors extracted by the encoder module are stored at both instance-level and category-level using the feature storage module. The feature storage module includes an instance-level feature storage module and a category-level feature storage module.

[0103] First, build an instance-level repository. To store the features of all images in the training set. Here, yes The i-th image feature. To ensure real-time updates of instance features, the current feature f is directly used in each iteration. i replace Where f i These are the sample features in the current batch of images B.

[0104] Then, build a category-level repository. This is used to store the proxy features of each category obtained through clustering algorithms in the training set, and to initialize the category proxy by selecting category centers within each category in each training epoch. Here, yes Having y i The proxy characteristics of tags.

[0105] To ensure real-time updates of the proxy features, in each iteration, the current feature y is directly sampled randomly. i The characteristics of the label f i replace Where f i These are the sample features in the current batch of images B.

[0106] Figure 8 The diagram illustrates a process for obtaining class-invariant and discriminative identification features according to an embodiment of this application.

[0107] like Figure 8 As shown, the specific implementation of the sample-to-instance learning module and the sample-to-category learning module in step 5) is as follows:

[0108] 1) Given a sample feature f in the current B i We have this feature f i Applying a contrastive learning process from samples to instances, allowing features f i Aligning with all features within the category in the instance-level feature store allows for full utilization of global information and helps overcome feature variations within the class.

[0109] The optimization objective is in the form of:

[0110]

[0111] in, s∈S, n is the number of samples in the training set. S is a set consisting of all positive samples, and f i They share the same pseudo-label, where τ is a temperature parameter.

[0112] 2) Given that y is present in the current B i A feature f of the category label i We have this feature f i Applying a contrastive learning process from samples to categories, allowing the current feature f to... i Proxy features belonging to the same category as those in the category-level feature store Proximity to, and distance from, other categories of agent features. This allows the learned features to be made more discriminative.

[0113] The optimization objective is in the form of:

[0114]

[0115] in, yes Having y i The proxy features of the label, where C is the number of categories.

[0116] Finally, see Figure 4 The network parameters are updated using backpropagation, combined with the sample-instance loss function. Sample-clustering loss function Perform model optimization;

[0117] The overall optimization objective is as follows:

[0118]

[0119] Where λ is the weighting parameter, which controls the proportion of different losses, and is set to 1.2 here.

[0120] Finally, Adam was chosen as the optimizer for the model, with a weight decay rate of 5×e-4, an initial learning rate of 3.5×10-4, and a decrease of 0.1 every 20 epochs compared to the previous value, for a total of 50 epochs.

[0121] The image recognition system, image recognition method, device and storage medium based on real-time features of this application are used to extract sample features from the input image data through the neural network of the encoder; based on the sample features, recognition features with class invariance and discriminativeness are obtained; and based on the recognition features, the image recognition result is obtained.

[0122] In summary, this application can update each cluster feature and category proxy feature in real time, preserving the original feature distribution and original data manifold in the feature category, and finally obtaining recognition features with class invariance and discriminativeness, which greatly improves the performance of the image recognition model and the accuracy of the recognition results.

[0123] Example 2

[0124] This embodiment provides an image recognition system that applies a real-time feature-based image recognition method. For details not disclosed in this embodiment, please refer to the specific implementation details of the real-time feature-based image recognition method in other embodiments.

[0125] Figure 9 The diagram shows a schematic representation of an image recognition system according to an embodiment of this application.

[0126] like Figure 9 As shown, the image recognition system of this application embodiment applies the image recognition method based on real-time features in embodiment 1, specifically including an encoder module 10, a feature recognition module 20, and an image recognition module 30.

[0127] Specifically,

[0128] Encoder module 10: Extracts sample features from the input image data using the encoder's neural network. Recognition feature module 20: Obtains class-invariant and discriminative recognition features based on the sample features. Image recognition module 30: Obtains image recognition results based on the recognition features. After preprocessing the pedestrian image to be detected, it is input into the trained model for encoding; finally, the retrieval and recognition results of the pedestrian image are output.

[0129] Figure 10 Another schematic diagram of the structure of an image recognition system according to an embodiment of this application is shown.

[0130] In a preferred embodiment, such as Figure 10 As shown, the image recognition system also includes an image preprocessing module 1.

[0131] The specific steps for preprocessing the training dataset include: First, resizing all training images in the dataset to 256×128; second, performing the following data augmentations on the resized images in sequence: random flipping, padding with 10 pixels, random cropping, and random erasing; finally, each mini-batch of image samples consists of 256 training images from 16 categories, and is sampled in a class-balanced manner before being input into the encoding module.

[0132] The feature recognition module 20 includes a sample-to-instance learning module 21 and a sample-to-category learning module 22. The sample-to-instance learning module is responsible for aligning the current sample features extracted by the encoder with the category features in the instance-level feature storage module, learning class-invariant features to mitigate intra-category feature variations for the same pedestrian. The sample-to-category learning module, on the other hand, requires sample features to be close to their category proxy features while moving away from other proxy category features, making the learned features more discriminative.

[0133] Before aligning sample features with their class features and aligning sample features with class proxy features via the feature recognition module 20, the system also includes a feature storage module. For example, Figure 10 As shown, the feature storage module includes a category-level feature storage module 23 and an instance-level feature storage module 24.

[0134] The category-level feature storage module and the instance-level feature storage module are responsible for storing the feature vectors extracted by the encoder module at the category level and instance level, respectively.

[0135] like Figure 10 As shown, the principle flow of the image recognition system in this embodiment mainly includes the following steps in the model training stage: Step 1) Input pedestrian image data into the network model through the input module. Step 2) Preprocessing module: Set preprocessing parameters to perform data augmentation on the input pedestrian image data. Step 3) Encoder module: Composed of a ResNet-50 neural network, responsible for extracting feature vectors from the preprocessed pedestrian images. This application does not limit the type of neural network; other neural networks such as VGG-16 and DenseNet can be used.

[0136] Step 4) The feature storage module performs category-level and instance-level feature storage on the feature vectors extracted by the encoder module. It consists of a category-level feature storage module and an instance-level feature storage module. The category-level and instance-level feature storage modules are respectively responsible for storing the category-level and instance-level features of the feature vectors extracted by the encoder module.

[0137] Step 5) Sample-to-Instance Learning Module and Sample-to-Category Learning Module. The sample-to-instance learning module is responsible for aligning the current sample features extracted by the encoder with the category features in the instance-level feature storage module, learning class-invariant features to mitigate intra-class feature variations for the same person. The sample-to-category learning module is responsible for requiring sample features to be close to their category proxy features while moving away from other proxy category features, making the learned features more discriminative.

[0138] Secondly, the testing phase mainly includes: preprocessing the pedestrian images to be detected, inputting them into the trained model for encoding; and finally outputting the retrieval and recognition results of the pedestrian images.

[0139] The image recognition system based on real-time features of this application extracts sample features from the input image data through the encoder module 10 and the encoder's neural network; the recognition feature module 20 obtains recognition features with class invariance and discriminative properties based on the sample features; and the image recognition module 30 obtains the image recognition result based on the recognition features.

[0140] In summary, this application can update each cluster feature and category proxy feature in real time, preserving the original feature distribution and original data manifold in the feature category, and finally obtaining recognition features with class invariance and discriminativeness, which greatly improves the performance of the image recognition model and the accuracy of the recognition results.

[0141] Example 3

[0142] This embodiment provides an image recognition device. For details not disclosed in the image recognition device of this embodiment, please refer to the specific implementation of the multi-mode image recognition method or system in other embodiments.

[0143] Figure 11 The diagram shows a structural schematic of an image recognition device 400 according to an embodiment of this application.

[0144] like Figure 11 As shown, the image recognition device 400 includes:

[0145] Memory 402: Used to store executable instructions; and

[0146] Processor 401: Used to connect to memory 402 to execute executable instructions to complete the motion vector prediction method.

[0147] Those skilled in the art will understand that the illustration Figure 11This is merely an example of an image recognition device 400 and does not constitute a limitation on the image recognition device 400. It may include more or fewer components than shown, or combine certain components, or different components. For example, the image recognition device 400 may also include input / output devices, network access devices, buses, etc.

[0148] The processor 401 (Central Processing Unit, CPU) can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or processor 401 can be any conventional processor. Processor 401 is the control center of the image recognition device 400, connecting all parts of the image recognition device 400 through various interfaces and lines.

[0149] The memory 402 can be used to store computer-readable instructions. The processor 401 implements various functions of the image recognition device 400 by running or executing the computer-readable instructions or modules stored in the memory 402 and by calling the data stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the image recognition device 400, etc. In addition, the memory 402 may include a hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, read-only memory (ROM), random access memory (RAM), or other non-volatile / volatile storage devices.

[0150] If the modules integrated in the image recognition device 400 are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by instructing related hardware through computer-readable instructions. The computer-readable instructions can be stored in a computer-readable storage medium, and when executed by a processor, the computer-readable instructions can implement the steps of the various method embodiments described above.

[0151] Example 4

[0152] This embodiment provides a computer-readable storage medium having a computer program stored thereon; the computer program is executed by a processor to implement the multi-modal image recognition method in other embodiments.

[0153] The image recognition device and storage medium of this application embodiment extract sample features from the input image data through the neural network of the encoder; obtain recognition features with class invariance and discriminativeness based on the sample features; and obtain image recognition results based on the recognition features.

[0154] In summary, this application can update each cluster feature and category proxy feature in real time, preserving the original feature distribution and original data manifold in the feature category, and finally obtaining recognition features with class invariance and discriminativeness, which greatly improves the performance of the image recognition model and the accuracy of the recognition results.

[0155] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0156] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0157] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0158] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0159] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular forms “a,” “the,” and “the” used in this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0160] It should be understood that although the terms first, second, third, etc., may be used in this invention to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of this invention, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0161] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0162] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. An image recognition method based on real-time features, characterized in that, Includes the following steps: Based on the input image data, sample features are extracted using a neural network-based encoder; Based on the sample features, align the sample features with the features of their respective categories, and learn class-invariant features; And by bringing the sample features close to the proxy features of their respective categories, discriminative features are learned; Obtain recognition features that are class-invariant and discriminative; Image retrieval results are obtained based on the identified features; The process of aligning the sample features with the features of their respective categories and learning class-invariant features includes: given one of the sample features; applying a sample-to-instance contrastive learning process to the feature to align the feature with all features of its respective category; Learning discriminative features by bringing the sample features closer to the surrogate features of their respective categories includes: given one of the sample features; applying a sample-to-category contrastive learning process to the feature, so that the feature is closer to the surrogate features of its respective category.

2. The image recognition method based on real-time features according to claim 1, characterized in that, The process involves aligning the sample features with the features of their respective categories, and learning class-invariant features. And by bringing the sample features close to the proxy features of their respective categories, discriminative features are learned; Prior to this, it also included: The sample features are stored as instance-level features to obtain the category features; the sample features are also stored as category-level features to obtain the category proxy features.

3. The image recognition method based on real-time features according to any one of claims 1-2, characterized in that, The step of extracting sample features from the input image data using an encoder-based neural network includes: The encoder model is pre-trained on the ImageNet dataset to obtain model weights; the encoder's neural network is then initialized based on these model weights. The feature layers extracted from the backbone network are subjected to generalized average pooling to obtain feature vectors, which are then standardized. Clustering algorithms are used to cluster all standardized features, and data belonging to the same cluster are assigned the same pseudo-label.

4. The image recognition method based on real-time features according to claim 2, characterized in that, The step of storing the sample features at the category level to obtain the category features includes: Construct an instance-level repository to store the features of all images in the training set; in each iteration, directly replace the feature corresponding to the instance-level repository with the current feature; A category-level repository is constructed to store the proxy features of each category obtained by clustering algorithm in the training set. In each training cycle, the category center proxy is selected in each category for initialization. In each iteration, the category proxy features corresponding to the category-level repository are replaced by random sampling.

5. An image recognition system based on real-time features, characterized in that, include: Encoder module: Used to extract sample features from the input image data through the encoder's neural network; Feature recognition module: used to align the sample features with the features of their respective categories based on the sample features, and learn class-invariant features; And by bringing the sample features close to the proxy features of their respective categories, discriminative features are learned; The process involves obtaining class-invariant and discriminative features. Specifically, aligning the sample features with features belonging to their respective categories to learn class-invariant features includes: providing one feature from the sample features; applying a sample-to-instance contrastive learning process to the feature to align it with all features belonging to its category; and moving the sample feature closer to a surrogate feature belonging to its category to learn discriminative features, which includes: providing one feature from the sample features; applying a sample-to-category contrastive learning process to the feature to move it closer to a surrogate feature belonging to its category. Image recognition module: used to obtain image retrieval results based on the recognition features.

6. An image recognition device, comprising: Memory: Used to store executable instructions; as well as Processor: For connection to memory to execute executable instructions to perform the image recognition method as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, It stores a computer program; the computer program is executed by a processor to implement the image recognition method as described in any one of claims 1-4.

Images