Training methods, application methods, equipment, and media for image recognition and retrieval models

Abstract

This application discloses a training method, application method, device, and medium for an image recognition retrieval model, belonging to the field of image recognition. The method includes: extracting visual modal features of seen categories; obtaining semantic modal information of seen and unseen categories, the semantic modal information including attribute modality, category name modality, and semantic description modality; fusing the semantic modal information to obtain an associative semantic representation; training a generative adversarial network based on the visual modal features and semantic representation of seen categories, and generating pseudo-visual modal features of unseen categories; training at least one of a classification model and a retrieval model based on the pseudo-visual modal features corresponding to unseen categories, such that the classification model at least identifies image samples of unseen categories, and / or that the retrieval model at least retrieves retrieval results matching unseen categories. This application improves the model's generalization ability and recognition accuracy under zero-sample conditions by introducing a category separation optimization mechanism.

Description

Technical Field

[0001] This application belongs to the field of image recognition, and in particular relates to a training method, application method, device and medium for an image recognition retrieval model. Background Technology

[0002] Image and video recognition and retrieval technologies, as an important branch of computer vision, have been widely deployed in various fields such as intelligent surveillance, human-computer interaction, autonomous driving, and sports analytics. With the increasing expressive power of visual models, higher demands are being placed on recognition and retrieval capabilities in open-category scenarios.

[0003] In related technologies, zero-shot learning methods have been proposed to address the problem of limited training data coverage. These methods mainly utilize the relevant information of seen categories available during the training phase to classify or retrieve images in the absence of image data for unseen categories. They often employ strategies such as feature mapping or generative models to complete training and inference.

[0004] However, the above methods still have shortcomings in practical applications, such as insufficient mining of semantic information and lack of structural constraints between generated features, which ultimately leads to insufficient generalization ability for the recognition of unseen categories. Summary of the Invention

[0005] This application aims to address at least one of the technical problems existing in related technologies. To this end, this application proposes a training method, application method, device, and medium for an image recognition retrieval model to improve recognition generalization ability and scalability, enabling efficient category discrimination and semantic retrieval even in the absence of unseen category image samples.

[0006] Firstly, this application provides a method for training an image recognition retrieval model, the method comprising:

[0007] Acquire sample images of the seen categories and extract the true visual modal features corresponding to the seen categories;

[0008] Semantic modal information corresponding to seen categories and semantic modal information corresponding to unseen categories are obtained respectively; wherein, the semantic modal information includes at least attribute modality, category name modality and semantic description modality, which are used to characterize the semantic association between different categories;

[0009] The semantic modal information of the seen category and the unseen category are fused separately to obtain the semantic representation of the association with the seen category and the semantic representation of the association with the unseen category.

[0010] Based on the real visual modal features and related semantic representations corresponding to the seen categories, a generative adversarial network is trained, and the trained generative adversarial network is used to process the related semantic representations corresponding to the unseen categories to obtain pseudo visual modal features corresponding to the unseen categories.

[0011] At least one of a classification model and a retrieval model is trained based on pseudo-visual modal features corresponding to unseen categories, such that the classification model at least identifies image samples of unseen categories, and / or that the retrieval model at least retrieves retrieval results matching unseen categories.

[0012] The training method of the image recognition retrieval model provided in this application has good recognition generalization ability and model scalability, and can achieve efficient category discrimination and semantic retrieval under the condition of lack of target category image samples.

[0013] Secondly, this application provides an image recognition retrieval method, the method comprising:

[0014] Acquire the image to be identified and extract the target visual modality features from the image to be identified;

[0015] Based on a pre-trained classification model, the visual modal features of the target are predicted to identify the target category contained in the image to be identified; and / or

[0016] Based on the pre-trained retrieval model, a retrieval is performed in the prototype vector set according to the target visual modality features to obtain retrieval results that match the image to be identified;

[0017] The image recognition retrieval model is obtained according to the training method of the image recognition retrieval model described in the first aspect.

[0018] According to the image recognition and retrieval method provided in this application, when recognizing and retrieving images to be recognized, it can support the recognition of target categories that have not been trained, even in the absence of image samples of unseen categories. It can accurately distinguish the target category in the image to be recognized by simply performing image feature extraction and model forward prediction operations, which significantly improves the adaptability and maintainability in dynamically expanded category scenarios and has strong open category recognition capabilities.

[0019] Thirdly, this application provides a training apparatus for an image recognition retrieval model, the apparatus comprising:

[0020] The first acquisition module is used to acquire sample images of the seen categories and extract the real visual modal features corresponding to the seen categories;

[0021] The first acquisition module is further configured to acquire semantic modal information corresponding to seen categories and semantic modal information corresponding to unseen categories, respectively; wherein, the semantic modal information is used to characterize the semantic association between different categories from at least the attribute modality, category name modality and semantic description modality;

[0022] The semantic module is used to fuse the semantic modal information of the seen category and the unseen category respectively to obtain the semantic representation of the association with the seen category and the semantic representation of the association with the unseen category.

[0023] The generation module is used to train a generative adversarial network based on the real visual modal features and related semantic representations corresponding to the seen categories, and to process the related semantic representations corresponding to the unseen categories through the trained generative adversarial network to obtain pseudo visual modal features corresponding to the unseen categories.

[0024] A training module is used to train at least one of a classification model and a retrieval model based on pseudo-visual modal features corresponding to unseen categories, so that the classification model can at least identify image samples of unseen categories, and / or the retrieval model can at least retrieve retrieval results that match unseen categories.

[0025] Fourthly, this application provides an image recognition and retrieval device, the device comprising:

[0026] The second acquisition module is used to acquire the image to be identified and extract the target visual modal features from the image to be identified.

[0027] An application module is used to predict the target visual modal features based on a pre-trained classification model to identify the target category contained in the image to be identified; and / or, based on a pre-trained retrieval model, to perform a retrieval in the prototype vector set according to the target visual modal features to obtain a retrieval result matching the image to be identified; wherein the image recognition retrieval model is obtained according to the training method of the image recognition retrieval model described in the first aspect.

[0028] Fifthly, this application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the training method for the image recognition retrieval model as described in the first aspect above, or the image recognition retrieval method as described in the second aspect.

[0029] In a sixth aspect, this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the training method for the image recognition retrieval model as described in the first aspect above, or the image recognition retrieval method as described in the second aspect.

[0030] In a seventh aspect, this application provides a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run computer programs or instructions to implement the training method of the image recognition retrieval model as described in the first aspect above, or the image recognition retrieval method as described in the second aspect.

[0031] Eighthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the training method for the image recognition retrieval model as described in the first aspect above, or the image recognition retrieval method as described in the second aspect.

[0032] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0033] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0034] Figure 1 This is a flowchart illustrating the training method of the image recognition retrieval model provided in some embodiments of this application;

[0035] Figure 2 This is a flowchart illustrating the steps of semantic association fusion provided in some embodiments of this application;

[0036] Figure 3 This is a flowchart illustrating the category separation optimization provided in some embodiments of this application;

[0037] Figure 4 This is a flowchart illustrating the image recognition and retrieval method provided in some embodiments of this application;

[0038] Figure 5 This is a flowchart illustrating the multimodal zero-shot identification and retrieval method provided in some embodiments of this application;

[0039] Figure 6 This is a schematic diagram of the structure of the training device for the image recognition retrieval model provided in some embodiments of this application;

[0040] Figure 7 This is a schematic diagram of the structure of the image recognition and retrieval device provided in some embodiments of this application;

[0041] Figure 8 This is a schematic diagram of the structure of a computer device provided in some embodiments of this application. Detailed Implementation

[0042] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0043] Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used in the description of this application is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms "comprising" and "having," and any variations thereof, in the description, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the description, claims, or accompanying drawings of this application are used to distinguish different objects, not to describe a specific order or hierarchy.

[0044] In this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, in this application, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0045] In this application, "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).

[0046] In related technologies, recognition and retrieval models largely rely on large-scale labeled data. However, in real-world applications, there are often numerous emerging or unseen categories (such as abnormal behaviors in emergencies or special movements in professional sports). Data for these categories is difficult to collect and label, leading to poor model performance when handling unseen categories. Furthermore, traditional recognition and retrieval methods typically focus on learning the direct mapping between visual modal features and category labels, failing to fully utilize the semantic information of the linguistic modality of the categories. This further limits the model's generalization ability under zero-shot conditions. Therefore, how to achieve efficient recognition and retrieval of unseen category samples through multimodal information fusion under data constraints has become one of the key research challenges.

[0047] Zero-shot recognition and retrieval are two typical tasks under zero-shot learning. Zero-shot recognition focuses on classifying video or image samples of unseen categories, aiming to accurately identify the semantic label of the input sample from categories not seen during training. Zero-shot retrieval, on the other hand, focuses on cross-modal semantic retrieval. It calculates and ranks the similarity between target features and category prototypes, returning the most similar categories as the retrieval results.

[0048] However, the zero-shot approach still has certain limitations. On the one hand, most related techniques rely solely on single semantic information, failing to integrate multiple semantic information such as language modality attributes, category names, and descriptive statements. They lack effective means to align multi-source semantic data using correlation analysis tools, making it difficult to align multi-source semantic information within a shared semantic space. This results in incomplete and inaccurate semantic representations of emerging / unseen categories, making it difficult to capture the deep semantic characteristics of categories under conditions of scarce multimodal data. On the other hand, in the process of visual modality feature generation, related techniques lack targeted optimization for the separability of generated feature categories and fail to effectively constrain them using statistical distribution information between and within categories. This leads to significant overlap between generated features from different categories in the new feature space, thus affecting the classifier's performance in classifying new category images or videos and in semantic retrieval.

[0049] Furthermore, related technologies primarily focus on the consistency between generated features and real visual modal features when generating features, while neglecting the possibility of improving the quality of generated features through statistical indicators such as scatter matrices. This results in insufficient generalization ability for invisible categories in multimodal zero-shot recognition and retrieval scenarios.

[0050] In view of this, embodiments of this application provide a training method for an image recognition and retrieval model and an image recognition and retrieval method. This method integrates a recognition mechanism that combines semantic fusion and category separability optimization. By fusing three types of semantic information—attributes, category names, and category descriptive statements—a more comprehensive semantic representation is constructed, thereby effectively transferring knowledge of common categories and enabling the recognition and retrieval of visual modal data not seen during the training phase. Furthermore, an optimization mechanism based on inter-category divergence and intra-category divergence is introduced into the Generative Adversarial Network (GAN) to improve the separability and diversity of generated features, thereby enhancing the model's recognition and retrieval performance under zero-sample conditions.

[0051] This application provides a training method for an image recognition and retrieval model, as well as an image recognition and retrieval method, applicable to the recognition and retrieval tasks of multimodal zero-shot images and videos, and particularly suitable for application scenarios where target categories are constantly expanding and training samples are difficult to obtain. Typical applications include abnormal behavior recognition, medical image-assisted diagnosis, industrial visual inspection, and remote monitoring.

[0052] The training method for the image recognition retrieval model provided in this application will be described in detail below with reference to the accompanying drawings and specific embodiments and application scenarios. The training method for the image recognition retrieval model provided in this application can be executed by a computer device or a functional module or entity within a computer device capable of implementing the training method for the image recognition retrieval model. The following description uses a computer device as an example to illustrate the training method for the image recognition retrieval model provided in this application.

[0053] Figure 1 This is a flowchart illustrating the training method of the image recognition and retrieval model provided in some embodiments of this application. For example... Figure 1 As shown, the training method of this image recognition retrieval model includes steps 110 to 120.

[0054] Step 110: Obtain sample images of the seen categories and extract the real visual modal features corresponding to the seen categories.

[0055] Here, sample images refer to image data containing category targets used for training or testing. Sample images can be pictures or video frames from videos.

[0056] In this context, "seen categories" (or common categories) refer to categories that have corresponding image samples and semantic information during the training phase. Typically, seen categories are image or video targets for which sample data is widely available in practical applications, such as daily behaviors, common objects, and general events. Conversely, "unseen categories" (or emerging categories, invisible categories) refer to categories that do not have image samples during the training phase but possess semantic information. Unseen categories typically include rare or special categories that do not appear in the training data, such as sudden events, professional actions, domain-specific objects, or unusual targets in rare scenes.

[0057] Image or video data of common categories can be obtained from publicly available datasets (such as ImageNet, Kinetics, UCF101, etc.) or by collecting data independently and extracting visual modal features through convolutional neural networks (such as ResNet, I3D, C3D, etc.).

[0058] Visual modal features refer to multi-dimensional feature vectors that can represent the content of an image. Real visual modal features are multi-dimensional vectors extracted from sample images by image coding networks, used to characterize the visual content of an image, such as spatial structure, texture, and semantics.

[0059] Computer equipment acquires image data of seen categories for training and extracts real visual modal features corresponding to the seen categories through an image coding network. The image coding network can employ a convolutional neural network (CNN) architecture, such as ResNet. For example, the visual modal features are represented as follows: Where d is the feature dimension, S represents the number of samples in the common categories, with the subscript S indicating a common category. It is important to emphasize that this refers to the set of known categories. With unseen category set Non-overlapping, that is .

[0060] Step 120: Obtain semantic modal information corresponding to the seen categories and semantic modal information corresponding to the unseen categories respectively; wherein, the semantic modal information includes at least attribute modality, category name modality and semantic description modality, which are used to characterize the semantic relationship between different categories.

[0061] Semantic modality information refers to textual or structured semantic data used to describe category features, typically derived from natural language expressions. Seen and unseen categories share the same semantic space for cross-category multimodal knowledge transfer.

[0062] In this embodiment, the semantic modality of semantic modality information includes, but is not limited to, attribute labels, category names, and semantic descriptions. Attribute modality refers to an attribute vector that structurally defines key features of a category (e.g., "has a tail," "can fly," etc.), involving the target's appearance, behavior patterns, scenarios, or interaction frequencies, and is typically constructed manually or through rules. For example, each attribute entry serves as a dimension of the attribute vector. Category, attribute vector The The attribute value of each position Indicates category With the The correlation of each attribute item. The attribute vectors of all categories constitute the attribute matrix. ,in For the number of categories, The dimension of the attribute vector.

[0063] Category name modality refers to the natural language names of categories, which are transformed into fixed-dimensional vectors using a word embedding model (such as word2vec). For example, the category name vectors of all categories constitute the category name semantic matrix. ,in The dimension of the class noun vector.

[0064] Semantic description modality refers to a complete natural language description of category features, represented using sentence embedding models (such as BERT). For example, the word vectors of all categories constitute the sentence semantic matrix. ,in This represents the dimension of the sentence word vector.

[0065] The computer device acquires semantic modal information corresponding to both seen and unseen categories. This semantic modal information includes attribute modality, category name modality, and semantic description modality, which represent the structural features, short name, and natural language description of the category, respectively. The attributes, category name, and semantic description of each category are represented by these semantic matrices, reflecting the semantic relationships between categories.

[0066] For example, for the "vehicle" category, its attribute modalities could include: "has wheels" and "has a steering wheel"; the category name is "car"; and the semantic description could include: "a mobile device that is powered by an internal combustion engine or battery to move and can change direction by a steering wheel." Similarly, for the "cat" category, its attribute modalities could include: "has four legs," "has fur," "meows," and "has whiskers"; the category name is "cat"; and the semantic description could include: "a small mammal, usually kept as a pet, with soft fur, sensitive hearing and vision, often meows, and has good night vision."

[0067] Step 130: Perform fusion processing on the semantic modal information of the seen category and the unseen category respectively to obtain the semantic representation of association corresponding to the seen category and the semantic representation of association corresponding to the unseen category.

[0068] Relevance semantic representation refers to a unified vector representation generated by fusing information from multiple semantic modalities. It guides the generation or recognition process of visual features by uniformly modeling and fusing multimodal semantic vectors such as attributes, names, and descriptions (e.g., canonical correlation analysis).

[0069] The computer device processes the semantic modal information of both seen and unseen categories separately to obtain semantic representations of their respective associations. Fusion methods may include strategies such as semantic mapping, feature concatenation, and spatial alignment.

[0070] For example, the computer device will display the attribute matrix. Class name semantic matrix and statement semantic matrix Mapping alignment is performed, and deep nonlinear associations are explored through kernelized canonical correlation analysis to obtain a semantic representation of association. It should be noted that the extraction and fusion processing of semantic modal information corresponding to each category can be performed using one or more neural networks, which can be called semantic networks. This application does not limit the specific structure of such semantic networks.

[0071] Step 140: Based on the real visual modal features and related semantic representations corresponding to the seen categories, train a generative adversarial network (GAN), and process the related semantic representations corresponding to the unseen categories through the trained GAN to obtain pseudo visual modal features corresponding to the unseen categories.

[0072] Generative Adversarial Networks (GANs) are neural network frameworks consisting of a generator and a discriminator, used to simulate data distributions.

[0073] A computer device trains a generative adversarial network (GAN) based on the real visual modal features of seen categories and their corresponding semantic representations, enabling the network to generate visual modal features that match the input semantic representations. After training, the computer device uses the GAN to process the semantic representations of unseen categories, obtaining pseudo-visual modal features for the unseen categories.

[0074] For example, a computer device constructs a WGAN (Wasserstein Generative Adversarial Nets) network. The input is a relational semantic representation and a Gaussian noise vector, and the output is a pseudo-visual feature with the same dimension as the visual feature. During training, a gradient penalty strategy is used, and training stops after 5000 iterations. The final output is used to generate feature samples for 5 unseen categories.

[0075] Step 150: Train at least one of a classification model and a retrieval model based on pseudo-visual modal features corresponding to the unseen category, so that the classification model can at least identify image samples of the unseen category, and / or, so that the retrieval model can at least retrieve retrieval results that match the unseen category.

[0076] The classification model can be, for example, a classifier used to perform image category discrimination tasks, while the retrieval model can be, for example, a matching model or a similarity measurement network used to perform image retrieval tasks. The former predicts the visual modal features of the target image and outputs the corresponding category label; the latter calculates the similarity between the target image features and the prototype vectors of each candidate category and outputs the K most relevant categories.

[0077] It should be noted that the specific implementation methods of classification and retrieval models are not limited to the examples above. They can be implemented using neural networks, graph models, Transformer structures, or other machine learning frameworks. The specific selection can be optimized and configured according to the application scenario and computing resources.

[0078] To achieve image recognition, the computer device constructs a training sample set based on pseudo-visual modal features and category labels corresponding to multiple unseen categories, and trains a classification model using supervised learning. Specifically, the classification model is a multi-class classifier, its structure including an input layer, multiple hidden layers, and an output layer. Each neuron in the output layer corresponds to an unseen category. During training, a cross-entropy loss function is used to minimize the difference between the model's predictions and the actual labels, and the model parameters are continuously optimized using a backpropagation algorithm. After training, the classification model is deployed to the image recognition and retrieval system. When a new image to be recognized is input, the computer device first extracts the visual modal features of the image and inputs them into the classification model. The model outputs the predicted probability distribution between these features and all learned unseen categories, and uses the category with the highest probability as the recognition result, thereby achieving accurate identification of image samples of unseen categories.

[0079] To achieve image retrieval, the computer device first calculates the pseudo-visual prototype vector for each unseen category using mean aggregation based on the pseudo-visual modal features corresponding to each unseen category, thus constructing a prototype vector set. This prototype vector set serves as the index foundation for the image retrieval system and is used for subsequent model training. After training, the retrieval model can be deployed in the image recognition and retrieval system. When a user uploads an image to be retrieved, the system extracts its visual modal features and maps them to the target space through an embedding network. Subsequently, the computer device calculates the visual similarity (such as cosine similarity or Euclidean distance) between these features and all unseen category prototype vectors, sorts them according to the similarity scores, and returns the image or category information corresponding to the most similar category, thereby achieving accurate retrieval of unseen category samples. By introducing a trainable similarity mapping mechanism based on the prototype vectors, the model can perceive fine-grained differences between pseudo-visual modal features, achieving high-precision retrieval of unseen categories without relying on real image samples, thus improving the model's flexibility and practical deployment value.

[0080] Therefore, an image recognition and retrieval model is constructed based on an image coding network, a semantic network, a generative adversarial network, a classification model, and / or a retrieval model. It should be noted that the image coding network, semantic network, generative adversarial network, classification model, and / or retrieval model can be trained jointly or separately at different stages; this application does not impose any restrictions on this.

[0081] Through the above steps, computer equipment can generate pseudo-visual modal features with class discriminative capabilities using semantic information, even without real image samples. This allows for the training of a classification model capable of recognizing or retrieving unseen categories. For example, for newly added unseen categories, pseudo-visual modal features can be generated based on their corresponding semantic modal information. Incremental training of the recognition and retrieval model using these pseudo-visual modal features enables the recognition of the new category. Compared to traditional methods, this training method significantly reduces the cost of model expansion and updates, decreases reliance on computational resources, and enhances the model's application capabilities in open-category scenarios by training the recognition and retrieval model solely to adapt to new categories.

[0082] The training method for the image recognition retrieval model provided in this application has good recognition generalization ability and model scalability, and can achieve efficient category discrimination and semantic retrieval under the condition of lack of target category image samples. Specifically, by acquiring sample images of seen categories and extracting real visual modal features, a realistic visual representation space can be constructed, providing reliable training samples for subsequent generative models. This effectively improves the generator's semantic-to-visual mapping accuracy and introduces semantic modal information of both seen and unseen categories. Furthermore, by introducing semantic modal information and fusing attribute modalities, category name modalities, and semantic description modalities to form associative semantic representations, a more complete and semantically discriminative category representation can be constructed, helping to alleviate feature confusion caused by insufficient category semantics. Training a generative adversarial network based on real visual features and semantic representations enhances the authenticity and diversity of generated features, strengthens the model's semantic perception at the image level, and provides a basis for generating reliable pseudo-visual samples in zero-shot scenarios. Subsequently, the trained generative model is used to generate features for unseen categories and train at least one of a classification model and a retrieval model. This enables the classification model to effectively classify images of unseen categories, and the retrieval model learns category prototypes to achieve distance-based image retrieval, significantly enhancing the model's application capability and flexibility in open-category scenarios.

[0083] Furthermore, the image recognition and retrieval model trained using the above method can support the recognition and retrieval of newly added unseen categories simply by updating the recognition and retrieval model. This eliminates the need to repeatedly train the entire model to expand the recognition capabilities, enabling the model to quickly adapt to new categories. This significantly reduces training costs and computational overhead, decreases dependence on hardware resources, and enhances the scalability and deployment flexibility of the model.

[0084] Furthermore, since the above model can be updated incrementally without loading a large number of training samples, and the recognition process is lightweight and efficient, it is suitable for deployment and use on edge computing devices or resource-constrained environments.

[0085] In related technologies, it is difficult to accurately capture the nonlinear relationships between various semantic modalities, resulting in limited discriminative power and expressiveness in the semantic space, which affects the modeling quality of category semantics. To address this, in some embodiments, the semantic modal information of seen and unseen categories is fused separately to obtain the semantic representation of associations corresponding to seen categories and the semantic representation of associations corresponding to unseen categories. This includes: performing nonlinear mapping on the semantic modal information of attribute modalities, category name modalities, and semantic description modalities corresponding to each category using kernel functions to obtain attribute kernel matrices, category name kernel matrices, and semantic description kernel matrices corresponding to each category; constructing an objective function for canonical correlation analysis based on the attribute kernel matrices, category name kernel matrices, and semantic description kernel matrices corresponding to each category; and obtaining the semantic representation of associations corresponding to each category by solving the objective function.

[0086] In this embodiment, semantic modality information is defined as category representation data composed of multiple semantic dimensions, including attribute modality, category name modality, and semantic description modality. The computer device performs fusion processing on the semantic modality information of seen and unseen categories, specifically including the following steps:

[0087] First, the computer device applies a kernel function to each type of semantic modality information for nonlinear mapping. The kernel function can be a radial basis function (RBF) or a polynomial kernel. After mapping, attribute kernel matrices, category name kernel matrices, and semantic description kernel matrices are obtained, which are used to capture the nonlinear structural features of each semantic dimension in high-dimensional space. For example, the computer device uses a kernel function... (For example, radial basis function kernels (RBF) can be used to implement nonlinear mappings to map the original matrix to a higher-dimensional feature space.)

[0088] For attribute matrix , mapped to ,in For the first Attribute vectors for each category This is the kernel matrix of the attribute matrix. Similarly, for the class name semantic matrix... , mapped to ,in For the first Class noun vectors for each category, This is the kernel matrix of the class name semantic matrix. For the statement semantic matrix... , mapped to ,in For the first The word vectors of each category of sentences, This is the kernel matrix of the semantic matrix of statements.

[0089] Next, the computer device uses three kernel matrices as input to construct a Canonical Correlation Analysis (CCA) objective function, which aims to maximize the correlation between different semantic modalities, explore the potential correspondence between attributes and language semantics, and transform the optimization problem into a generalized eigenvalue problem.

[0090] To maximize the three kernel matrices , and The correlation between them, and the objective function for constructing canonical correlation analysis of computer devices:

[0091]

[0092] in, , as well as These are the kernel matrix , and The projection coefficient, yes The transpose of , and the rest are similar.

[0093] Finally, by optimizing the objective function, the computer device obtains the optimal projection coefficient vectors for each semantic modality. The three projected semantic modality vectors are then concatenated to form the associative semantic representation of the corresponding category. This representation serves as the input to subsequent generative and classification models. For example, if the solution to this optimization problem is obtained through generalized eigenvalue decomposition, the computer device calculates the covariance matrix of each pair of matrices, and then calculates the eigenvalues ​​and eigenvectors of the covariance matrix to obtain the optimized projection coefficient vectors. (Associative semantic representation) It can be calculated as follows:

[0094]

[0095] Where C is the number of categories, and q is the spatial dimension of the semantic representation of association. This indicates a concatenation operation. Relational semantic representation. It contains deep nonlinear semantic structures between categories, providing semantic prior support for subsequent feature generation and discrimination.

[0096] In the above embodiments, the fusion strategy of kernel mapping and canonical correlation analysis effectively integrates three semantic information modalities: attribute modality, category name modality, and semantic description modality, solving the problems of scattered semantic expression, inconsistent dimensions, and weak correlation. Compared with linear concatenation or mean fusion, it can significantly improve the expressiveness and class discriminativeness of semantic representation, providing a more accurate and robust semantic foundation for subsequent feature generation and recognition, and enhancing the model's generalization ability to unseen categories.

[0097] In the process of fusing information from multiple semantic modalities, although canonical correlation analysis (CCA) can be used to capture the correlation between semantic modalities, relying solely on the original kernel matrix to construct the objective function makes it difficult to fully utilize the statistical structural information between the modalities. Therefore, in some embodiments, the semantic representation of association corresponding to each category is obtained by solving the objective function. This includes: for any category, calculating the covariance matrix of each semantic modality based on the corresponding attribute kernel matrix, category name kernel matrix, and semantic description kernel matrix, and calculating the cross-modal covariance matrix between every two semantic modalities; performing generalized eigenvalue decomposition based on each covariance matrix and each cross-modal covariance matrix to obtain the target projection coefficients corresponding to each semantic modality; projecting the kernel matrix corresponding to each semantic modality based on the target projection coefficients, and concatenating the projection results to obtain the semantic representation of association corresponding to the target category.

[0098] The computer device first defines the covariance matrix as the self-distribution structure of any semantic modality in a high-dimensional kernel space, used to characterize the range of variation of samples within that modality in the feature dimension. The cross-modal covariance matrix refers to the joint variation structure between any two semantic modalities under the same category, used to capture the statistical dependencies between different modalities.

[0099] For each category, the computer device calculates the covariance matrix of the attribute kernel matrix, the category name kernel matrix, and the semantic description kernel matrix, and calculates the cross-modal covariance matrix for pairwise combinations of the three modalities (attribute-name, attribute-description, name-description). Then, based on these covariance matrices and the cross-modal covariance matrix, the computer device constructs a generalized eigenvalue decomposition problem and performs the generalized eigenvalue decomposition operation to obtain the optimal projection coefficients that maximize the correlation between each modality, i.e., the projection vector corresponding to each semantic modality.

[0100] Finally, the computer device uses the obtained projection coefficients to perform linear projection on the corresponding kernel matrix, mapping each semantic modality to a unified projection space, and concatenates the projected vectors to generate the final associative semantic representation.

[0101] In the above embodiments, the introduction of covariance matrices and cross-modal covariance matrices enables explicit modeling of the distribution structure between semantic modalities during semantic fusion, effectively improving the consistency and discriminativeness of semantic projection. Generalized eigenvalue decomposition, as the core optimization method, makes the fused semantic representation more structurally separable and expressively powerful, significantly improving the adaptability of both the generative and classification models to semantic input. Compared to traditional fusion methods based solely on kernel matrices, this method enhances the distinguishability between categories while preserving semantic integrity, making it suitable for zero-shot recognition scenarios with more complex semantic differences.

[0102] In zero-shot recognition and retrieval tasks, semantic modality information, as a priori representation at the category level, serves as a crucial bridge connecting visual space and category labels. In some embodiments, acquiring semantic modality information corresponding to seen categories and semantic modality information corresponding to unseen categories includes: acquiring multiple attribute entries corresponding to each category, and obtaining semantic modality information corresponding to attribute modalities for each category based on the correlation between the attribute entries; inputting the category names of each category into a first language processing model to obtain semantic modality information corresponding to the category name modality for each category; and inputting the semantic descriptions of each category into a second language processing model to obtain semantic modality information corresponding to the semantic description modality for each category.

[0103] For attribute modalities, the computer device selects multiple relevant attribute items for each category based on a preset set of attributes, and determines the correlation value between the attribute items and the category through manual scoring, expert rules, or statistical correlation, thus forming an attribute vector.

[0104] For category name modalities, the computer device inputs each category name into a first language processing model, which can be, for example, a word-level embedding model (such as word2vec, GloVe, etc.) and outputs word vectors of fixed dimensions, thereby using them as semantic modal information of the category name modalities.

[0105] For semantic description modalities, the computer device inputs the description sentences corresponding to each category into a second language processing model. This second language processing model can be, for example, a sentence-level semantic encoder (such as BERT, Sentence-BERT, etc.) and outputs semantic embedding vectors, which are then used as semantic modal information of the semantic description modalities.

[0106] The first language model and the second semantic model can be the same or different natural language processing models, and this application does not impose any restrictions on this.

[0107] Ultimately, the three semantic modalities constitute the semantic basis representation for each category, serving as input for subsequent relational semantic fusion processing.

[0108] Figure 2 This is a flowchart illustrating the steps of semantic association fusion provided in some embodiments of this application. For example... Figure 2 As shown, the computer device receives three types of semantic modal information: attributes, category names, and semantic descriptions (corresponding to the category descriptions in the figure), and constructs a unified associative semantic representation based on these three types of semantic information.

[0109] For attribute modalities, the computer device acquires attribute modal information and constructs the corresponding attribute matrix. The computer device applies a kernel function based on this attribute matrix. Construct the kernel matrix corresponding to the attribute modes To enhance its nonlinear expressive power.

[0110] For category name modalities, the computer device performs natural language processing (e.g., word2vec embedding) on ​​the category name modal information for each category to construct a category name semantic matrix. Similarly, through kernel functions Perform kernel mapping on it to obtain the category name kernel matrix. .

[0111] For semantic description modalities, computer devices use natural language processing models (such as BERT or Sentence-BERT) to convert the text descriptions of categories into semantic embedding vectors, constructing a sentence semantic matrix. And through kernel function Obtain the corresponding kernel matrix .

[0112] Next, the computer device performs projection operations on the three kernel matrices respectively to obtain the projected vectors: Attribute Projection: Category Name Projection: Semantic description projection: ;in, , and The projection coefficient vectors, optimized through canonical correlation analysis (CCA) or its kernelized version (KCCA), are used to map the kernel matrix to the shared association space. Finally, the computer concatenates the three projection results to obtain the final semantic representation of the association. , where q represents the dimension of the concatenated vector. This semantic representation retains the core features of each semantic modality and completes multimodal fusion in a shared space, which can be used for subsequent pseudo-visual modality feature generation or recognition and discrimination tasks.

[0113] In the above embodiments, by standardizing the acquisition method of semantic modality information, a unified and semantically rich multi-source semantic input vector can be constructed efficiently and accurately, avoiding the problem of inconsistent semantic information acquisition paths between different categories, and effectively improving the consistency and fusion quality of the semantic space. Furthermore, using attribute labels, name word embedding, and sentence semantic encoding to acquire semantic modality information not only enhances the expressiveness of category semantics but also significantly improves the model's ability to model complex semantic structures, providing a solid semantic foundation for subsequent feature generation and classification.

[0114] As mentioned earlier, in zero-shot learning tasks, generative adversarial networks (GANs) have become the mainstream data augmentation technique for generating pseudo-visual modal features of unseen categories. Related technologies often focus on the overall consistency between the generator output and the real visual distribution, but fail to precisely optimize for the specific structure of the real visual features of already seen categories, easily leading to blurry generated features and poor category discrimination.

[0115] Therefore, in some embodiments, a generative adversarial network (GAN) is trained based on real visual modal features and related semantic representations corresponding to the seen categories. This includes: inputting real visual modal features corresponding to the seen categories into the GAN to be trained to obtain pseudo visual modal features corresponding to the seen categories; constructing an overall loss function based on the pseudo visual modal features corresponding to the seen categories and the extracted real visual modal features corresponding to the seen categories; and training the GAN to be trained based on the overall loss function until the training termination condition is met, thus obtaining a trained GAN.

[0116] Generative Adversarial Networks (GANs) are adversarial learning structures consisting of a generator and a discriminator. The generator generates visual modal features based on the input semantic information, while the discriminator determines whether the input features are real or generated samples.

[0117] The computer device first inputs the semantic representation of the association with the seen categories as a conditional input, along with a noise vector, into the generator to be trained, generating pseudo-visual modal features for the corresponding categories. These features should simulate the distribution and structure of real visual modal features. Subsequently, the computer device inputs the generated pseudo-visual modal features and the real visual modal features corresponding to the category into the discriminator, and constructs an overall loss function based on the difference between the two. This overall loss function is jointly constructed based on a generative adversarial loss function and a two-divergence loss function, used to jointly optimize the feature generation capability and inter-class discrimination capability of the generative adversarial network. Specifically, the generative adversarial loss function establishes an adversarial training mechanism between the generator and the discriminator, prompting the generator to output pseudo-visual modal features with a distribution close to that of real visual modal features; the two-divergence loss function, by introducing a divergence loss mechanism, guides the generator to improve the separability of features while preserving class discriminability, thereby improving the overall quality and class discriminability of the generated pseudo-visual modal features.

[0118] For example, generative adversarial loss functions may include Wasserstein distance loss, mean squared error loss, or other losses, used to measure the closeness between generated features and real features. The details of the two-divergence loss function will be discussed later.

[0119] Finally, based on the overall loss function, the computer equipment uses optimization algorithms such as gradient descent to alternately train the generator and discriminator until the overall loss function converges or the training termination conditions such as the preset number of training rounds or error threshold are reached, thus completing the training process of the generative adversarial network.

[0120] In the above embodiments, by using real visual modal features as a reference benchmark for adversarial supervision, the realism and stability of generated features can be significantly improved during the training of the generative adversarial network. Simultaneously, a loss function incorporating both real and pseudo-features is constructed, enabling the generator to learn category structure information more fully during optimization, thus preventing generation collapse or distortion. This method effectively improves the consistency of pseudo-visual modal features in distribution matching and category discrimination, providing a high-quality data foundation for subsequent training, recognition, and retrieval tasks, and enhancing the model's generalization ability and reliability.

[0121] During the training of generative adversarial networks (GANs), relying solely on the overall adversarial loss between generated and real features often fails to adequately characterize the discriminative structure between categories. This can easily lead to confusion in the distribution of pseudo-visual modal features from different categories within the feature space, affecting the accuracy of subsequent recognition and retrieval. To enhance the class discriminative ability of pseudo-features, it is necessary to introduce structured statistical metrics into the loss function, particularly by measuring the dispersion between categories and the concentration within categories to guide generator optimization.

[0122] To this end, in some embodiments, a loss function is constructed based on the pseudo-visual modal features corresponding to the seen categories and the extracted real visual modal features corresponding to the seen categories. This includes: for any seen category, mapping the corresponding pseudo-visual modal features to a relaxation vector through a fully connected layer, and calculating the mean vector for the seen category based on the relaxation vector; determining the global mean vector based on the relaxation vectors of each seen category; constructing inter-class divergence loss and intra-class divergence loss based on the mean vectors of each seen category and the global mean vector, respectively; and constructing a dual divergence loss function with the goal of maximizing the inter-class divergence loss and minimizing the intra-class divergence loss.

[0123] First, the computer device maps pseudo-visual modal features to a vector space, obtaining relaxation vectors. This transformation is typically achieved using a fully connected layer to ensure a uniform distribution scale for the features. For any known category, the computer device inputs all its pseudo-visual modal features into a fully connected mapping layer, resulting in a set of relaxation vectors of uniform dimension. The mean of this set is then calculated as the category mean vector, representing the category's central position in the feature space. Next, based on the relaxation vectors of all known categories, the computer device calculates the overall distribution center, obtaining a global mean vector, which measures the average position of all category features in the space. Furthermore, the computer device defines the dispersion between each category's mean vector and the global mean vector as inter-class divergence, and the closeness of each sample within the same category to the category mean as intra-class divergence, constructing inter-class divergence losses and intra-class divergence losses respectively. Finally, the computer device optimizes the objective function by maximizing the inter-class divergence loss and minimizing the intra-class divergence loss, constructing a composite loss function to guide the generator to enhance the separability of generated features from different categories while preserving semantic consistency. To distinguish it, the loss function constructed here based on inter-class divergence loss and intra-class divergence loss is called the bi-divergence loss function. For example, this bi-divergence loss function can be a weighted sum of inter-class divergence loss and intra-class divergence loss.

[0124] On the other hand, the computer device inputs the semantic representation of the association with the seen categories into the generator to generate corresponding pseudo-visual modality features, while simultaneously extracting the real visual modality features corresponding to the category, and inputting both into the discriminator. The discriminator is used to determine whether the input features originate from real samples, thereby outputting the discrimination probability. During training, the generator aims to generate features that are difficult for the discriminator to identify as forgeries, even if the pseudo-visual modality features are as close as possible to the real visual modality features; while the discriminator aims to accurately distinguish between real features and generated features. Based on this adversarial objective, a generative adversarial loss function is constructed, allowing the generator and discriminator to continuously optimize during the training process, thereby improving the authenticity and diversity of pseudo-visual modality features.

[0125] Therefore, the computer device can jointly train the generative adversarial network (GAN) to be trained based on the divergence loss function and the generative adversarial loss function. This allows the generator to learn and generate pseudo-visual modal features that are as close as possible to the distribution of real visual modal features, thus "deceiving" the discriminator. Simultaneously, it enables the discriminator to distinguish between the generated pseudo-visual modal features and real visual modal features, thereby improving discrimination accuracy. Through this mutually reinforcing and competitive adversarial relationship, the GAN learns representations that are closer to the real feature distribution during continuous optimization, thereby improving the ability of subsequent classification or retrieval models to identify and retrieve unseen categories.

[0126] Figure 3 This is a schematic diagram of the category separation optimization process provided in some embodiments of this application. For example... Figure 3 As shown in the embodiments of this application, a random noise vector can be introduced. ,in This represents a Gaussian distribution with a mean of 0 and a variance of 1. The noise vector... semantic representation of association with the corresponding category splicing to form a generator Input, generator Generate pseudo-visual modality features based on input. Discriminator For the generated pseudo-visual modal features and the true visual modal features of this category The goal of this differentiation is to maximize the difference between the scores of real samples and generated samples. Following the WGAN training method, its loss function... Defined as:

[0127]

[0128] in, Expressing expectations, This represents the true visual modality feature vectors of common categories. Simultaneously, the discriminator undergoes weight clipping or gradient penalty to satisfy the Lipschitz continuity condition.

[0129] generator The goal is to minimize the following generator loss (i.e., the generative adversarial loss function) ensures that the generated visual modal features are as close as possible to the distribution of real visual modal features, and can prevent the discriminator from misclassifying them as real data:

[0130]

[0131] However, discriminator consistency constraints alone cannot guarantee that generated features will have good discriminative ability among new categories. To ensure that generated visual modal features are not only realistic but also have good separability, category separability constraints are also needed. First, the generator's pseudo-visual modal features... Mapped to a relaxation vector through a fully connected layer. :

[0132]

[0133] in, It is a fully connected mapping layer. For the mapping matrix, This is the bias term. This mapping provides a certain relaxation effect, preventing difficulties in generator training. The relaxed vector after mapping... Perform category-level statistical analysis. For Category, number of samples Its mean vector is defined as: .in, belong The vector at the j-th position of the category. The global mean is defined as: .in, , This represents the number of categories.

[0134] Inter-class scatter matrix The following formula measures the degree of dispersion of samples between categories:

[0135]

[0136] Intraclass scatter matrix The following formula measures the dispersion among samples within the same category:

[0137]

[0138] To enhance the separability of generated features, it is desirable to maximize inter-class divergence and minimize intra-class divergence in the new feature space. A projection matrix is ​​introduced. The objective function is: Taking all factors into account, a class separation constraint is introduced into the generator's optimization objective, resulting in an overall loss function. This can be represented as a generative adversarial loss function. With the two divergence loss function The weighted sum, i.e.:

[0139]

[0140] in, These are the weight parameters used to balance the WGAN loss (i.e., the generative adversarial loss function). ) and category separation constraint The relationship between (i.e., the two divergence loss function).

[0141] During the training process, the generator is first trained using WGAN. Given noise and related semantic representation Under the given conditions, realistic visual modal features are generated. Next, the generated features are mapped to a new feature space through a fully connected layer, and the between-class and within-class scatter matrices of the mapped features are calculated. Finally, the overall loss is minimized. The generator and projection matrix are optimized to ensure the realism of the generated visual modal features while providing better class separation, thereby improving the performance of multimodal zero-shot recognition and retrieval.

[0142] Generative Adversarial Network (WGAN) training and class separability optimization are employed to effectively simulate and structurally enhance visual modal features of emerging or invisible categories. Training utilizes the visual modal features of common categories and their semantic associations with linguistic modalities. Adversarial learning generates pseudo-visual modal features consistent with semantics. Based on this, class separability optimization maximizes inter-class divergence and minimizes intra-class divergence, ensuring the class separability of generated features in the new feature space. Ultimately, this improves the classification and retrieval models' ability to identify and retrieve emerging or invisible categories.

[0143] In the above embodiments, by introducing intra-class divergence and inter-class divergence as optimization metrics for the loss function, the structural separability of pseudo-visual modal features in the feature space is significantly enhanced. During the training of the generative adversarial network, not only is the consistency between pseudo-features and real features improved, but also the spatial boundaries between different categories are clearly constrained through divergence optimization, effectively reducing inter-category overlap and improving the recognition accuracy and robustness of images of unseen categories in subsequent recognition and retrieval tasks. This method maintains the stability of the generative model while improving discriminative performance and is suitable for large-scale category expansion and open recognition environments.

[0144] In zero-shot image recognition and retrieval tasks, pseudo-visual modal features, serving as alternative training samples for unseen categories, are a crucial foundation for zero-shot learning in classification and retrieval models. Traditional classifiers or retrieval models are typically trained solely on pseudo-samples for supervision, which can lead to classification boundary shifts or underfitting, especially when pseudo-sample quality fluctuates or category distributions are imbalanced. Therefore, it is necessary to set flexible loss functions for different training needs: on the one hand, in scenarios involving unseen categories, model training can be based solely on pseudo-features; on the other hand, when simultaneously handling seen and unseen categories, real visual features must be incorporated into the training process to construct more comprehensive discriminative supervision, thereby enhancing the model's generalization ability to samples from different category sources.

[0145] Therefore, in some embodiments, at least one of a classification model and a retrieval model is trained based on pseudo-visual modal features corresponding to unseen categories, so that the classification model can at least identify image samples of unseen categories, and / or, so that the retrieval model can at least retrieve retrieval results matching unseen categories. This includes: constructing a first classification loss function based on pseudo-visual modal features and category labels corresponding to unseen categories, or constructing a second classification loss function based on pseudo-visual modal features and category labels corresponding to unseen categories, as well as real visual modal features and pseudo-visual modal features corresponding to seen categories; and training the classification model based on the first or second classification loss function so that the classification model can identify image samples of unseen categories, or simultaneously identify image samples of both unseen and seen categories.

[0146] In the first scenario, to identify only image samples belonging to unseen categories, the computer device constructs a training sample set using pseudo-visual modal features and their category labels corresponding to several unseen categories, and builds a first classification loss function based on this training set. This loss function can be cross-entropy loss, used to measure the accuracy of the classification model in predicting the category to which the pseudo-visual modal features belong. Subsequently, the classification model is trained using supervised learning methods, enabling it to receive visual modal feature inputs from real images during the testing phase and output probability distributions corresponding to each unseen category, thereby identifying the unseen category to which the target image belongs.

[0147] In the second scenario, to enable the model to simultaneously recognize images of both seen and unseen categories, the computer equipment incorporates real and pseudo-visual modal features related to seen categories into the training sample set, and integrates these features with pseudo-visual modal features of unseen categories during training. A second classification loss function is then constructed to comprehensively constrain the model's classification performance across seen and unseen categories. During training, the model simultaneously learns the feature differences and category boundaries between real and pseudo-visual modalities, improving generalization ability. After training, this classification model can take the visual modal features of any test image as input and output the predicted probability of it belonging to any seen or unseen category, achieving recognition tasks within a unified category space. Thus, the trained recognition and retrieval model can not only identify whether an image contains seen or unseen categories, but also output specific category labels, such as vehicle, cat, dog, table, etc.

[0148] By employing the training strategy described above, an image recognition model with high accuracy and generalization ability can be constructed without relying on real image samples of unseen categories. This significantly improves the adaptability and scalability of the zero-shot recognition system and reduces the model's dependence on manually labeled samples.

[0149] For example, the semantic information of unseen category language modalities, after being processed by the aforementioned semantic network, yields an associative semantic representation matrix. ,in The number of invisible categories, This is the spatial dimension for representing the semantic associations. For the categories to be identified and retrieved... Its corresponding semantic representation vector is denoted as Using a pre-trained Generative Adversarial Network (WGAN), under conditional generation mode, semantic representations of unseen categories are generated. With random noise vector As input, pseudo-visual modal features for this category are generated. The generation process can be represented as:

[0150]

[0151] in, For the generated visual modality features, The dimension representing the visual modality features. For each unseen category, multiple samplings are performed. A set of generative visual modal features can be constructed. ,in The number of samples generated for this category. A new training dataset is created by combining the pseudo-visual modal features obtained from the generator for each unseen category with their category labels:

[0152]

[0153] in, Invisible category The corresponding tags.

[0154] For unseen categories, utilize the dataset. Training classifier Its training objective is to minimize the classification loss function. That is, the first classification loss function:

[0155]

[0156] in, This represents cross-entropy loss or other suitable loss functions.

[0157] During the testing phase, given an image or video to be identified, visual modal features are extracted using the aforementioned convolutional neural network. Input it into the pre-trained classifier Perform category discrimination:

[0158]

[0159] in, This indicates the predicted label corresponding to the image or video.

[0160] In another instance, the computer device can also assemble a pseudo-visual modality training set by combining the pseudo-visual modality features obtained by the generator for each emerging / invisible category with their category labels:

[0161]

[0162] in, Invisible category The corresponding tags The set of invisible categories. The set of real samples for visible categories is represented as:

[0163]

[0164] in, Visible category The corresponding tags This is the set of visible categories. To achieve a unified classification task for visible and invisible categories, the set of real samples for visible categories needs to be... With the generated set of pseudo-visual samples Merge and build a hybrid dataset:

[0165]

[0166] Therefore, computer devices can train classifiers based on mixed datasets.

[0167] In category recognition, the computer device utilizes the aforementioned hybrid dataset. Train a unified classification model The goal of training this system is to simultaneously identify both visible and invisible categories, with the objective of minimizing the classification loss function. That is, the second classification loss function:

[0168]

[0169] in, This represents cross-entropy loss or other suitable loss functions.

[0170] During the testing phase, given an image or video to be identified, visual modal features are extracted using the aforementioned convolutional neural network. Input it into the pre-trained classifier Perform category discrimination:

[0171]

[0172] in, This indicates the predicted label corresponding to the image or video.

[0173] In the above embodiments, by introducing a flexible classification model training path, training strategies can be efficiently configured according to different application scenarios: if the goal is to identify only unseen categories, the use of training data can be minimized, and a low-resource classifier can be quickly deployed; if the goal is to process both seen and unseen categories simultaneously, real and pseudo features can be combined for joint training to enhance the overall discriminative ability and robustness of the model. This method not only improves the classifier's adaptability to complex category spaces but also significantly enhances the recognition coverage and practical deployment flexibility in zero-shot scenarios.

[0174] In other embodiments, training at least one of a classification model and a retrieval model based on pseudo-visual modal features corresponding to unseen categories, so that the classification model at least identifies image samples of unseen categories, and / or so that the retrieval model at least retrieves retrieval results matching unseen categories, further includes: calculating pseudo-visual prototype vectors corresponding to unseen categories based on pseudo-visual modal features corresponding to unseen categories, and / or constructing visual prototype vectors corresponding to seen categories based on real visual modal features corresponding to seen categories; using at least one of the pseudo-visual prototype vectors and visual prototype vectors as a prototype vector set; the prototype vector set is used to support the retrieval model in performing image retrieval operations and outputting retrieval results matching unseen categories and / or seen categories.

[0175] This prototype vector set is required for building an image retrieval model using computer equipment, and supports the retrieval model in performing image retrieval operations for unseen categories and / or seen categories. This prototype vector set can be used to support the model in performing image retrieval for unseen categories without requiring real image samples, effectively solving the problem that traditional image retrieval methods cannot retrieve images when there are no samples or insufficient samples for new categories.

[0176] Specifically, for each unseen category, the computer device, based on the previously trained generative adversarial network, obtains a set of pseudo-visual modal features corresponding to that category, and aggregates the features in this set, for example, by calculating the mean vector or using cluster center representation, thereby generating a pseudo-visual prototype vector corresponding to that unseen category. This pseudo-visual prototype vector can serve as the representation of the target category in visual space.

[0177] For a seen category, the computer device can directly utilize the real visual modal features corresponding to that category in the training set to construct a visual prototype vector for that category in the same way as for unseen categories (e.g., feature mean or cluster centers). This visual prototype vector is used to represent the visual center features of the seen category.

[0178] The computer device merges the aforementioned pseudo-visual prototype vectors with at least one of the visual prototype vectors to form a prototype vector set, which serves as the core query base for the image retrieval model.

[0179] In the actual retrieval process, after an image to be retrieved is input, the computer device extracts its visual modal features and calculates visual similarity (e.g., cosine similarity or Euclidean distance) between it and each prototype vector in the prototype vector set. Based on the similarity scores, the retrieval results of one or more categories that are most similar to the visual features of the image are output.

[0180] This scheme can achieve accurate matching of unseen category samples while retaining compatibility support for seen categories. It can construct category representations for retrieval without relying on real image samples, which significantly improves the system's generalization ability and deployment flexibility.

[0181] For example, in category retrieval, in generating and building datasets Based on this, a visual modal prototype for each category is calculated, serving as the query anchor point during retrieval. Specifically, for visible categories... Visual modal prototypes can be directly constructed using its real samples. Let there be a total of [number] samples in its training set. The set of visual features of each sample is: The prototype is calculated as follows:

[0182]

[0183] For invisible categories The set of pseudo-visual features it generates is The pseudo-visual prototype vector of the visual modality of this category is obtained by averaging the generated features of this category:

[0184]

[0185] in, Indicates category The corresponding pseudo-visual prototypes of the visual modalities. From this, a unified set of category prototypes can be obtained:

[0186]

[0187] During the testing phase, given an image or video, its visual modal features are first extracted using a convolutional neural network. Then, similarity (e.g., cosine similarity) is calculated with the visual prototypes of all candidate categories in the category prototype set, and with the category... The similarity score of the corresponding pseudo-visual prototype is calculated as follows:

[0188]

[0189] in, For similarity calculation functions, commonly used metrics include cosine similarity, Euclidean distance, or Mahalanobis distance.

[0190] In the above embodiments, by constructing pseudo-visual prototype vectors based on pseudo-visual modal features of unseen categories and visual prototype vectors based on real visual modal features of seen categories, and merging the two to form a prototype vector set, the generalization ability and practicality of the zero-shot image retrieval model can be significantly improved. Simultaneously, by combining real visual information of seen categories, the model's recognition accuracy for known categories can be guaranteed, enabling unified retrieval processing for both seen and unseen categories, improving the overall system's compatibility and scalability, and meeting the image recognition and retrieval needs in large-scale, open-category scenarios.

[0191] Based on the same inventive concept, this application also proposes an image recognition retrieval method, which is based on the image recognition retrieval model trained in the above embodiments to realize the category recognition of the detection objects contained in the image.

[0192] Figure 4 This is a flowchart illustrating the image recognition and retrieval method provided in some embodiments of this application. For example... Figure 4 As shown, the image recognition and retrieval method includes steps 410 to 420. Wherein:

[0193] Step 410: Obtain the image to be identified and extract the target visual modality features from the image to be identified;

[0194] Step 420: Based on the pre-trained classification model, predict the target visual modal features to identify the target category contained in the image to be identified; and / or, based on the pre-trained retrieval model, perform a retrieval in the prototype vector set according to the target visual modal features to obtain retrieval results that match the image to be identified.

[0195] The trained classification model and / or retrieval model are obtained according to the training method of the image recognition retrieval model involved in any of the above embodiments or combinations thereof. Specific steps can be referred to the aforementioned embodiments and will not be repeated here. The classification model must at least have the ability to identify unseen categories, and the retrieval model must at least have the ability to retrieve retrieval results matching unseen categories. Depending on the scenario, the classification model and / or retrieval model can focus on identifying unseen categories or simultaneously consider the identification and retrieval of both seen and unseen categories. In this embodiment, target visual modal features refer to visual feature vectors used to characterize the content of the image to be identified, which are typically extracted through an image coding model.

[0196] After acquiring the image to be recognized, the computer device processes it using the same image coding network as in the training phase to extract the target visual modality features corresponding to the image. Then, the computer device inputs the extracted target visual modality features into the model. The model performs category discrimination based on the internally established visual-semantic mapping relationship and outputs the target category label or semantic category identifier that best matches the input image, thus completing the identification of the target category contained in the image to be recognized.

[0197] The prediction operation can include classification, matching, retrieval, etc. The final output is a label representing the semantic category to which the image belongs, or the category name corresponding to the top K prototypes most similar to the target image in the retrieval task. The label may be an unseen category for which no real image was provided during the training phase.

[0198] According to the image recognition and retrieval method provided in this application, when recognizing and retrieving images to be recognized, it can support the recognition of target categories that have not been trained, even in the absence of image samples of unseen categories. It can accurately determine the target category in the image to be recognized by simply performing image feature extraction and model forward prediction operations, which significantly improves the adaptability and maintainability of the system in dynamically expanded category scenarios and has strong open category recognition capabilities.

[0199] In some embodiments, the target visual modal features are predicted based on a pre-trained classification model to identify the target category contained in the image to be identified. This includes: predicting the target visual modal features using the classification model to obtain the predicted probabilities of the target visual modal features and each candidate category; the candidate categories include seen categories and unseen categories that have been learned during the training of the classification model; and based on the predicted probabilities corresponding to each candidate category, the candidate category with the highest probability is taken as the target category corresponding to the target visual modal features.

[0200] First, the computer device converts the image to be recognized into target visual modal features and inputs these features into a classification model. Based on its learned class discrimination ability, the model calculates the matching score or predicted probability between the target visual modal features and each candidate category. The candidate categories include seen categories that participated in learning from real samples during the training phase, and unseen categories that participated in training through pseudo-visual modal features.

[0201] Subsequently, the computer device obtains the predicted probability corresponding to each candidate category, and selects the category with the highest predicted probability as the final discrimination result of the visual modality feature of the target, that is, it is identified as the target category corresponding to the candidate category.

[0202] For example, suppose the classification model learns three known categories during the training phase: "airplane," "car," and "train," and two unseen categories: "hot air balloon" and "submarine." A computer device acquires an image to be identified and extracts its visual modal features. The model calculates the predicted probabilities between these features and the five candidate categories: airplane 0.10, car 0.15, train 0.08, hot air balloon 0.60, and submarine 0.07. Based on the highest probability value, the computer device determines the target category of the image as "hot air balloon."

[0203] In the above embodiments, by performing unified prediction probability calculation on the target visual modal features and candidate categories, the fusion discrimination of seen and unseen categories under the same recognition path is realized. It can automatically identify the category of the input image without prior category filtering, simplifying the deployment logic and improving the flexibility and accuracy of the recognition system.

[0204] In zero-shot image retrieval tasks, traditional methods typically rely on text descriptions or tags as query terms, making effective matching difficult when the target image has not yet been seen. This embodiment, however, constructs a prototype vector set and uses category-level pseudo-visual modal features as visual query anchors, enabling computer devices to directly perform cross-category retrieval based on visual similarity between images.

[0205] Therefore, in some embodiments, based on a pre-trained retrieval model, a search is performed in the prototype vector set according to the target visual modality features to obtain retrieval results matching the image to be identified. This includes: obtaining the prototype vector set; calculating the visual similarity between the target visual modality features and each prototype vector in the prototype vector set using the retrieval model to obtain a similarity score corresponding to each candidate category; the prototype vector set includes pseudo-visual prototype vectors corresponding to unseen categories, and / or visual prototype vectors corresponding to seen categories; and based on the similarity scores, selecting multiple target categories similar to the image to be identified from the prototype vector set to obtain retrieval results matching the image to be identified.

[0206] First, the computer device invokes a pre-trained image recognition retrieval model and loads the prototype vector corresponding to each candidate category. The prototype vector refers to the representative visual features of the category obtained during the training phase through generative models or statistical analysis of real samples, serving as a benchmark for retrieval matching. Multiple prototype vectors form a prototype vector set, covering all seen categories and modeled unseen categories. Next, the computer device extracts the target visual modal features of the image to be retrieved and calculates the visual similarity between these features and each prototype vector in the prototype vector set. Similarity calculation can use methods such as Euclidean distance, cosine similarity, or Mahalanobis distance to obtain a similarity score corresponding to each candidate category. Finally, the computer device sorts the candidate categories from highest to lowest based on the similarity scores and returns the category names corresponding to the top K prototypes most similar to the target image from the prototype set based on the sorting results, forming a candidate retrieval result set and outputting it.

[0207] In the above embodiments, by constructing a category prototype vector set and performing visual similarity matching, cross-category semantic retrieval can be completed directly based on image features without the need for real image labels or text queries. This method combines the model's ability to model unseen categories with an efficient matching mechanism in the vector space, which not only improves the compatibility of the retrieval model in open environments but also lowers the deployment threshold, making it suitable for fast and intuitive image retrieval applications in large-scale image systems.

[0208] Figure 5 This is a flowchart illustrating the multimodal zero-shot identification and retrieval method provided in some embodiments of this application. For example... Figure 5 As shown, firstly, the computer device performs the training dataset construction operation. This dataset includes image samples from known categories and their corresponding semantic modal information, including attribute modalities, category name modalities, and semantic description modalities, used to describe the semantic features of each category. Furthermore, semantic information from unseen categories can be introduced to support knowledge transfer in zero-shot scenarios.

[0209] Next, the computer device performs semantic association processing, which specifically includes: performing high-dimensional embedding on the three semantic modalities of attributes, category names, and semantic descriptions respectively, generating a kernel matrix through kernel function mapping, and further using the kernelized canonical correlation analysis method to model and fuse the deep nonlinear associations between the three modalities to obtain the associative semantic representation.

[0210] Subsequently, the computer equipment is trained using a generative adversarial network (GAN) and combined with a class-separating optimization mechanism, enabling the generator to possess stronger class discrimination and generalization abilities during the process of learning the semantic-to-visual mapping. During training, the generator uses semantic representations as conditional inputs and combines them with random noise to generate pseudo-visual modal features; the discriminator is used to distinguish between real visual features and generated features, and optimizes the distribution structure of generated features by introducing structural loss functions for inter-class and intra-class divergence.

[0211] After training the generative adversarial network (GAN), the computer device uses the GAN to generate pseudo-visual modal features for emerging / unseen categories. For each unseen category, a set of pseudo-visual modal features is generated using associative semantic representations and random noise as input, forming a pseudo-sample dataset that can be used for training or matching.

[0212] In the application phase, computer devices can perform two types of operations based on the generated pseudo-visual modality features:

[0213] (1) Image category recognition: Input the visual modal features of the image to be recognized into the trained classifier and output the corresponding target category label, supporting the direct recognition of unseen categories;

[0214] (2) Image category retrieval: The visual modal features of the target image are matched with the prototype vectors of pseudo features of each category, and the names of the top K most similar categories are returned based on the results, thus realizing cross-category retrieval.

[0215] Finally, the identification or retrieval results can be used for tasks such as automatic annotation, behavior analysis, and resource archiving in practical application scenarios, and the process ends.

[0216] Therefore, by deeply fusing multi-source semantic information and employing kernelized canonical correlation analysis to map and align the semantic information of various language modalities, including attributes, category names, and descriptive statements, a comprehensive and associative semantic representation is obtained. This fully explores the nonlinear relationships between different semantic sources, providing comprehensive and refined semantic guidance for subsequent generative models, thus significantly enhancing the effective transfer capability of common category knowledge to emerging or invisible categories. Secondly, category separability optimization is achieved through generative adversarial networks (GANs). During the generation process of the GAN, an optimization mechanism based on inter-class and intra-class divergence is introduced to project constraints on the generated visual modal features. The divergence matrix is ​​used to maximize the inter-class differences and minimize intra-class fluctuations of the generated features, ensuring higher class separability of the generated visual modal features in the new feature space. This reduces the overlap of features between categories and effectively alleviates the feature confusion problem in new category identification and retrieval. Furthermore, the aforementioned multi-source semantic fusion and category separability optimization techniques are organically integrated into a zero-shot recognition and retrieval framework. The generator generates visual modal features under the combined influence of input semantics and noise, and the distribution of generated features is optimized using a discriminator and divergence matrix constraints. This framework can effectively identify and retrieve invisible categories in image or video scenes without relying on real visual samples of new categories, and has good generalization and scalability.

[0217] Based on the same inventive concept, embodiments of this application also provide a training method for an image recognition retrieval model, wherein the executing entity can be a training device for the image recognition retrieval model. This application embodiment uses an image recognition retrieval model training device executing the image recognition retrieval model training method as an example to illustrate the image recognition retrieval model training device provided in this application embodiment.

[0218] This application also provides a training device for an image recognition retrieval model, which is applied to a computer device. Figure 6 This is a schematic diagram of the structure of a training device for an image recognition and retrieval model provided in some embodiments of this application. For example... Figure 6 As shown, the training device for this image recognition and retrieval model includes an acquisition module 601, a semantic module 602, a generation module 603, and a training module 604. Wherein:

[0219] The first acquisition module 601 is used to acquire sample images of the seen categories and extract the real visual modal features corresponding to the seen categories.

[0220] The first acquisition module 601 is further configured to acquire semantic modal information corresponding to the seen categories and semantic modal information corresponding to the unseen categories, respectively; wherein the semantic modal information is used to characterize the semantic association between different categories from at least the attribute modal, category name modal and semantic description modal.

[0221] The semantic module 602 is used to fuse the semantic modal information of the seen category and the unseen category respectively to obtain the semantic representation of the association with the seen category and the semantic representation of the association with the unseen category.

[0222] The generation module 603 is used to train a generative adversarial network based on the real visual modal features and related semantic representations corresponding to the seen categories, and to process the related semantic representations corresponding to the unseen categories through the trained generative adversarial network to obtain pseudo visual modal features corresponding to the unseen categories.

[0223] Training module 604 is used to train at least one of a classification model and a retrieval model based on pseudo-visual modal features corresponding to unseen categories, such that the classification model at least identifies image samples of unseen categories, and / or that the retrieval model at least retrieves retrieval results matching unseen categories.

[0224] The training apparatus for the image recognition retrieval model in this application embodiment can be a computer device or a component within the computer device, such as an integrated circuit or a chip. The computer device can be a terminal device or a server. The training apparatus for the image recognition retrieval model provided in this application embodiment can implement the various processes and achieve the same technical effects as the various embodiments of the above-described image recognition retrieval model training method; to avoid repetition, further details are omitted here.

[0225] Based on the same inventive concept, this application also provides an image recognition and retrieval device, the execution subject of which can be a computer device. Figure 7 This is a schematic diagram of the structure of an image recognition and retrieval device provided in some embodiments of this application. For example... Figure 7 As shown, the image recognition and retrieval device includes a second acquisition module 701 and an application module 702. Wherein:

[0226] The second acquisition module 701 is used to acquire the image to be recognized and extract the target visual modal features in the image to be recognized.

[0227] Application module 702 is used to predict the visual modal features of a target based on a pre-trained classification model to identify the target category contained in the image to be identified; and / or, based on a pre-trained retrieval model, to perform a retrieval in the prototype vector set according to the visual modal features of the target to be identified, to obtain retrieval results matching the image to be identified. The classification model and retrieval model are based on, for example... Figure 1 The training method for the image recognition and retrieval model shown is obtained.

[0228] The image recognition retrieval device in this application embodiment can be a computer device or a component within a computer device, such as an integrated circuit or a chip. The computer device can be a terminal device or a server. The image recognition retrieval device provided in this application embodiment can implement the various processes implemented in the above-described image recognition retrieval methods and achieve the same technical effects; to avoid repetition, further details are omitted here.

[0229] Figure 8 This is a schematic diagram of the structure of a computer device provided in some embodiments of this application. In some embodiments, such as Figure 8 As shown, this application embodiment also provides a computer device 800, including a processor 801, a memory 802, and a computer program stored in the memory 802 and executable on the processor 801. When the program is executed by the processor 801, it implements the various processes of the above-described method embodiments and can achieve the same technical effects. To avoid repetition, it will not be described again here.

[0230] This application also provides a non-transitory computer-readable storage medium storing a computer program. When executed by a processor, this computer program implements the various processes of the above-described image recognition retrieval model training method or image recognition retrieval method embodiment, achieving the same technical effect. To avoid repetition, further details are omitted here. The processor is the processor in the computer device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0231] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described image recognition retrieval model training method or image recognition retrieval method.

[0232] This application also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described image recognition retrieval model training method or image recognition retrieval method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here. It should be understood that the chip mentioned in this application embodiment can also be called a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0233] The above are merely preferred embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for training an image recognition retrieval model, characterized in that, The method comprises: acquiring sample images of seen categories and extracting real visual modality features corresponding to the seen categories; respectively acquiring semantic modality information corresponding to the seen categories and semantic modality information corresponding to unseen categories; wherein the semantic modality information at least includes attribute modality, category name modality and semantic description modality, and is used to represent semantic correlation between different categories; respectively performing fusion processing on the semantic modality information of the seen categories and the unseen categories to obtain correlation semantic representations corresponding to the seen categories and correlation semantic representations corresponding to the unseen categories; training a generative adversarial network based on the real visual modality features corresponding to the seen categories and the correlation semantic representations, wherein the process of training the generative adversarial network comprises: mapping pseudo visual modality features output by a generator to a new feature space, and determining a category mean vector of the pseudo visual modality features of each seen category in the new feature space to obtain a category center, and determining a global mean vector to measure the overall distribution center of all category features in the new feature space; determining an inter-class scatter loss used to represent the separation between categories according to the discrete degree between the category center corresponding to each seen category and the overall distribution center, and determining an intra-class scatter loss used to represent the concentration within the same category according to the close degree between the generated features within the same category and the corresponding category center; constructing a bi-scatter loss function based on the inter-class scatter loss and the intra-class scatter loss, the bi-scatter loss function being used to constrain the distribution structure of the generated features of different categories in the feature space to reduce the overlap between the features of different categories; jointly optimizing the bi-scatter loss function and a generative adversarial loss function of the generative adversarial network to obtain a trained generative adversarial network, and processing the correlation semantic representation corresponding to the unseen categories through the trained generative adversarial network to obtain pseudo visual modality features corresponding to the unseen categories; training at least one of a classification model and a retrieval model based on the pseudo visual modality features corresponding to the unseen categories, so that the classification model at least identifies image samples of the unseen categories, and / or so that the retrieval model at least retrieves retrieval results matching the unseen categories.

2. The method of claim 1, wherein, The fusion processing on the semantic modality information of the seen categories and the unseen categories respectively to obtain the correlation semantic representations corresponding to the seen categories and the correlation semantic representations corresponding to the unseen categories comprises: nonlinearly mapping the semantic modality information of the attribute modality, the category name modality and the semantic description modality corresponding to each category through a kernel function to obtain an attribute kernel matrix, a category name kernel matrix and a semantic description kernel matrix corresponding to each category respectively; constructing an objective function for canonical correlation analysis based on the attribute kernel matrix, the category name kernel matrix and the semantic description kernel matrix corresponding to each category respectively; obtaining the correlation semantic representations corresponding to each category by solving the objective function.

3. The method of claim 2, wherein, The obtaining of the correlation semantic representations corresponding to each category by solving the objective function comprises: For any category, based on the corresponding attribute kernel matrix, category name kernel matrix and semantic description kernel matrix, the covariance matrix of each semantic modality itself is calculated, and the cross-modal covariance matrix between each two semantic modalities is calculated; Based on each covariance matrix and each cross-modal covariance matrix, generalized eigenvalue decomposition is performed to obtain target projection coefficients corresponding to each semantic modality; Based on the target projection coefficients, the kernel matrix corresponding to each semantic modality is projected, and each projection result is spliced to obtain the relevance semantic representation corresponding to the category.

4. The method according to any one of claims 1 to 3, characterized in that, The respective acquisition of the semantic modality information corresponding to the seen category and the semantic modality information corresponding to the unseen category comprises: Obtain a plurality of attribute entries corresponding to each category respectively, and obtain the semantic modality information corresponding to the attribute modality of each category based on the correlation between each attribute entry; The category name of each category is input into the first language processing model to obtain the semantic modality information corresponding to the category name modality of each category; The semantic description of each category is input into the second language processing model to obtain the semantic modality information corresponding to the semantic description modality of each category.

5. The method of claim 1, wherein, The training of at least one of the classification model and the retrieval model based on the pseudo visual modality feature corresponding to the unseen category, so that the classification model at least identifies the image sample of the unseen category, and / or so that the retrieval model at least retrieves the retrieval result matched with the unseen category, comprises: Based on the pseudo visual modality feature corresponding to the unseen category and the category label corresponding to the unseen category, a first classification loss function is constructed, or based on the pseudo visual modality feature corresponding to the unseen category and the category label corresponding to the unseen category, and the real visual modality feature and the pseudo visual modality feature corresponding to the seen category, a second classification loss function is constructed; Based on the first classification loss function or the second classification loss function, the classification model is trained, so that the classification model can identify the image sample of the unseen category, or simultaneously identify the image sample of the unseen category and the seen category.

6. The method of claim 1, wherein, The training of at least one of the classification model and the retrieval model based on the pseudo visual modality feature corresponding to the unseen category, so that the classification model at least identifies the image sample of the unseen category, and / or so that the retrieval model at least retrieves the retrieval result matched with the unseen category, further comprises: Based on the pseudo visual modality feature corresponding to the unseen category, a pseudo visual prototype vector corresponding to the unseen category is calculated, and / or based on the real visual modality feature corresponding to the seen category, a visual prototype vector corresponding to the seen category is constructed; At least one of the pseudo visual prototype vector and the visual prototype vector is used as a prototype vector set; the prototype vector set is used to support the retrieval model to perform image retrieval operation and output the retrieval result matched with the unseen category and / or the seen category.

7. An image recognition search method characterized by, The method comprises: Obtaining a to-be-identified image and extracting a target visual modality feature in the to-be-identified image; Based on the pre-trained classification model, the target visual modality feature is predicted to identify the target category contained in the to-be-identified image; and / or retrieve, based on the target visual modality feature, in a prototype vector set, to obtain a retrieval result matched with the to-be-identified image; The classification model and the retrieval model are obtained according to the training method of the image recognition retrieval model in any one of claims 1 to 6.

8. The method of claim 7, wherein, The classification model and the retrieval model are obtained according to the training method of the image recognition retrieval model in any one of claims 1 to 6. The classification model and the retrieval model are obtained according to the training method of the image recognition retrieval model in any one of claims 1 to 6. The classification model and the retrieval model are obtained according to the training method of the image recognition retrieval model in any one of claims 1 to 6.

9. The method of claim 7, wherein, The classification model and the retrieval model are obtained according to the training method of the image recognition retrieval model in any one of claims 1 to 6. The classification model and the retrieval model are obtained according to the training method of the image recognition retrieval model in any one of claims 1 to 6. The classification model and the retrieval model are obtained according to the training method of the image recognition retrieval model in any one of claims 1 to 6. The classification model and the retrieval model are obtained according to the training method of the image recognition retrieval model in any one of claims 1 to 6.

10. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The classification model and the retrieval model are obtained according to the training method of the image recognition retrieval model in any one of claims 1 to 6.

11. A non-transitory computer-readable storage medium, comprising: The classification model and the retrieval model are obtained according to the training method of the image recognition retrieval model in any one of claims 1 to 6. The classification model and the retrieval model are obtained according to the training method of the image recognition retrieval model in any one of claims 1 to 6. The classification model and the retrieval model are obtained according to the training method of the image recognition retrieval model in any one of claims 1 to 6. The classification model and the retrieval model are obtained according to the training method of the image recognition retrieval model in any one of claims 1 to 6. The classification model and the retrieval model are obtained according to the training method of the image recognition retrieval model in any one of claims 1 to 6. The classification model and the retrieval model are obtained according to the training method of the image recognition retrieval model in any one of claims 1 to 6. The classification model and the retrieval model are obtained according to the training method of the image recognition retrieval model in any one of claims 1 to 6.

Images