An image recognition method and device

Abstract

The application provides an image recognition method and device, and relates to the technical field of image classification and recognition. In the disclosed scheme, the multi-stage feature decoupling and modal alignment are guided based on word vectors, effectively relieving the problems of visual feature entanglement and semantic combination ambiguity. Specifically, first, the visual feature decoupling in the image is realized through the channel attention and spatial attention mechanisms, and the attribute and object semantic information interaction enhancement in the image is realized through the bidirectional guiding mechanism. Further, the correlation between the multiple feature pairs is optimized by constructing the positive and negative sample pairs of multiple perspectives, and the effective alignment of the global visual feature and the semantic feature is realized. Furthermore, by balancing the object and attribute classification loss, the alignment loss and the combined classification loss, the collaborative optimization of the feature decoupling, the modal alignment and the combined prediction is realized, thereby improving the generalization ability of the image recognition prediction model to complex and various unknown combinations and improving the accuracy of the image recognition.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, and in particular to an image recognition method and apparatus. Background Technology

[0002] Combinatorial zero-shot learning (CZSL) aims to identify unseen combinations by acquiring knowledge from known attribute-object pair combinations. Previous works measured compatibility scores by directly mapping global image and semantic features to a common embedding space; however, this direct mapping fails to effectively address the problems of visual feature entanglement and semantic ambiguity caused by context dependencies. The high coupling between visual features of attributes and objects in images severely limits the generalization ability of CZSL to unseen combinations.

[0003] To address this challenge, recent researchers have explored strategies such as primitive representation learning, context-aware combinatorial modeling, cue-based learning using visual language models, and higher-order reasoning using hybrid architectures. Primitive representation learning focuses on extracting independent attribute and object information from mixed visual signals. For example, OADIS separates visual representations of similarity and difference by comparing image pairs sharing the same concept; SCEN utilizes contrastive learning to shape highly discriminative attribute and object prototypes. To achieve more thorough decoupling, some studies have introduced attention mechanisms. For instance, DRANet designed an inverse attention strategy that actively suppresses the feature response region of a concept, thereby strengthening its representation relative to the concept. Other studies argue that attributes and objects depend on visual cues at different levels and decouple them through architectural design. KG-SP constructs independent classifiers and feature extractors for attributes and objects, decoupling features based on the principle that global features are more applicable to objects, while local patterns are more important for attributes. CoT (HPL) designed object and attribute expert methods to extract object and attribute representations separately, achieving feature separation. In contrast to the decoupling approach, context-aware combinatorial modeling assumes that attributes are deeply dependent on the objects they are combined with. Therefore, its core lies in dynamically modeling the context of features, rather than eliminating dependencies. For example, CANet argues that there are no static, universal attribute representations and utilizes dynamically generated specific attribute embedding vectors based on object information to capture context-related visual details. In recent years, with the rapid development of Visual Modeling (VLM) text-to-image understanding capabilities, many works have transformed the combinatorial recognition problem into cue engineering. For instance, CSP treats attributes and objects as learnable soft cue lexical units, achieving zero-shot combinatorial reasoning by constructing a text description rich in combinatorial information that can be effectively understood by the VL model.

[0004] To overcome the limitations of single strategies, some studies have begun to integrate multiple information sources or reasoning paths, such as hybrid architectures like multimodal decoupling and fusion, multi-path integration architectures, and semantic enhancement. Multimodal decoupling leverages the inherent combinability of textual modalities to guide visual decoupling; for example, DFSP first decomposes combinatory concepts into independent attributes and objects in the text space, then guides visual features to align and fuse them. Multi-branch integration improves robustness by processing different tasks in parallel; for instance, Troika constructs independent attribute, object, and combinator recognition branches and fuses the prediction results of all three during decision-making. Furthermore, some works introduce external knowledge for semantic enhancement. For example, KG-SP uses knowledge graphs to evaluate the rationality of combinations, eliminating unreasonable combinations in an open-world setting; CDS-CZSL introduces the linguistic concept of "specificity," dynamically adjusting prediction weights by learning the rarity of attributes relative to objects, achieving higher-order semantic reasoning.

[0005] Despite significant progress made by existing methods, challenges remain. For example, pure vision methods are still insufficient in handling fine-grained semantics, while VLM methods may ignore key visual details due to over-reliance on pre-trained priors. Current CZSL methods face the dilemma of either focusing on context modeling while neglecting generalization or pursuing decoupling while losing key contextual information. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide an image recognition method and apparatus that overcomes the problems of high entanglement in visual feature space and semantic ambiguity by decoupling and aligning visual features and semantic features in multiple stages, thereby improving the accuracy of image recognition.

[0007] To address the aforementioned technical problems, embodiments of the present invention provide an image recognition method, comprising:

[0008] A word vector-guided image recognition prediction model is determined, which includes: a text feature entanglement module, a visual feature representation module, an adaptive feature decoupling module, and a loss function module;

[0009] The text word vectors of the training images in the training set are embedded into the text feature entanglement module, and interactive fusion processing is performed on the text word vector embeddings based on a preset guidance mechanism to obtain the interactively fused text semantic features; the text word vector embeddings include attribute word vector embeddings, object word vector embeddings, and combined word vector embeddings, and the text semantic features include attribute semantic features, object semantic features, and combined semantic features;

[0010] The global visual information corresponding to the training images in the training set is input into the visual feature representation module, and the global visual information is decoupled based on a preset attention mechanism to obtain local visual features in the training images. The local visual features include attribute visual features, object visual features, and combined visual features.

[0011] The text semantic features and the local visual features are input into the adaptive feature decoupling module for alignment and decoupling processing to obtain the mutual information loss value of the attribute-object combination in the training image; the mutual information loss value represents the degree of information sharing between attributes and objects.

[0012] In the loss function module, based on the similarity between the corresponding text semantic features and the local visual features, the preset cross-entropy loss function and the preset mutual information loss function, the total loss function is determined, and the learning parameters in the text feature entanglement module, the weight parameters in the adaptive feature decoupling module and the combined word vector embedding are updated according to the backpropagation mechanism of the total loss function. The above training process is repeated until the total loss function converges, so as to obtain the trained image recognition prediction model based on word vector and the prediction recognition mutual information loss value of attribute-object combination.

[0013] The image to be classified is input into a trained image recognition model for recognition processing, and the recognition result is obtained.

[0014] In one embodiment, the text word vector embedding of the training image is obtained by encoding the attribute text, object text, and combined text in the training image based on a pre-trained word embedding encoder.

[0015] In one embodiment, text word vectors from training images in the training set are embedded into the text feature entanglement module, and interactive fusion processing is performed on the text word vector embeddings based on a preset guidance mechanism to obtain interactively fused text semantic features, including:

[0016] Based on a preset attribute-guided embedding mechanism, the attribute word vector embeddings are weighted and fused into the object word vector embeddings to obtain the semantic features of the object;

[0017] Based on a preset object-guided embedding mechanism, the object word vector embeddings are weighted and fused into the attribute word vector embeddings to obtain the attribute semantic features;

[0018] The semantic features of the object, the semantic features of the attribute, and the combined word vector embedding are correlated to obtain the combined semantic features.

[0019] In one embodiment, global visual information corresponding to training images in the training set is input into the visual feature representation module, and the global visual information is decoupled based on a preset attention mechanism to obtain local visual features in the training images, including:

[0020] The input training image is mapped based on a preset visual encoder to obtain the global visual information of the training image.

[0021] Based on the preset attention mechanism, feature extraction is performed on the global visual information to obtain the local visual features.

[0022] In one embodiment, feature extraction is performed on the global visual information based on the preset attention mechanism to obtain the local visual features, including:

[0023] The global visual information is assigned to the object visual information and the attribute visual information, and combined visual information is obtained based on the object visual information and the attribute visual information.

[0024] The attribute visual information is used to extract features based on a preset spatial attention mechanism to obtain the attribute visual features.

[0025] The visual information of the object is extracted based on a preset channel attention mechanism to obtain the visual features of the object;

[0026] The combined visual information is feature extracted based on a preset multi-layer perception mechanism to obtain the combined visual features.

[0027] In one embodiment, the text semantic features and the local visual features are input into the adaptive feature decoupling module for alignment and decoupling processing to obtain the mutual information loss value of the attribute-object combination in the training image, including:

[0028] The text semantic features and the local visual features are paired to form the input sample pair of the adaptive feature decoupling module. The input sample pair includes text semantic feature sample pairs and local visual feature sample pairs.

[0029] The adaptive feature decoupling module performs concatenation mapping on the samples in the input sample pair to obtain the joint sample distribution and marginal sample distribution of the input sample pair;

[0030] Based on the joint sample distribution and the edge sample distribution, the mutual information loss value of the attribute-object combination in the training image is determined.

[0031] In one embodiment, the image recognition method further includes:

[0032] Based on preset semantic relationships, the text semantic features and the local visual features are grouped to obtain positive sample pairs and negative sample pairs. The positive sample pairs contain multiple text semantic feature sample pairs with a first correlation relationship and multiple local visual feature sample pairs with a second correlation relationship. The negative sample pairs contain multiple text semantic feature sample pairs with a third correlation relationship and multiple local visual feature sample pairs with a fourth correlation relationship. The first correlation relationship is greater than the third correlation relationship, and the second correlation relationship is greater than the fourth correlation relationship.

[0033] In one embodiment, determining the mutual information loss value of the attribute-object combination in the training image based on the joint sample distribution and the edge sample distribution includes:

[0034] The Gaussian kernel value between sample pairs in the joint sample distribution and the marginal sample distribution is determined based on a preset Gaussian kernel function, so as to quantify the difference between sample pairs in the joint sample distribution and the marginal sample distribution;

[0035] Based on the Gaussian kernel value and the preset loss weight for positive sample pairs, the first mutual information loss value of the attribute-object combination in the positive sample pair is determined;

[0036] Based on the Gaussian kernel value and the preset negative sample pair loss weight, the second recognition mutual information loss value of the attribute-object combination in the negative sample pair is determined.

[0037] In one embodiment, the total loss function is expressed as:

[0038] ;

[0039] in, , , Both represent the preset cross-entropy loss, used to measure the difference between the model's prediction and the true label; This represents the preset mutual information loss function obtained based on the mutual information loss value of attribute-object combination. , The first weight used to balance the attribute-object combined loss with the independent attribute or object loss. The second weight represents the loss of an independent attribute or object. This represents a regularization term used to balance the contribution of the mutual information loss value of attribute-object combinations to the total loss.

[0040] Embodiments of the present invention also provide an image recognition device, comprising:

[0041] The model building module is used to determine the image recognition prediction model based on word vector guidance. The image recognition prediction model includes: a text feature entanglement module, a visual feature representation module, an adaptive feature decoupling module, and a loss function module.

[0042] The model training module is used to input the text word vector embeddings of training images in the training set into the text feature entanglement module, and perform interactive fusion processing on the text word vector embeddings based on a preset guidance mechanism to obtain interactively fused text semantic features; the text word vector embeddings include attribute word vector embeddings, object word vector embeddings, and combined word vector embeddings, and the text semantic features include attribute semantic features, object semantic features, and combined semantic features; global visual information corresponding to the training images in the training set is input into the visual feature representation module, and the global visual information is decoupled based on a preset attention mechanism to obtain local visual features in the training images, the local visual features include attribute visual features, object visual features, and combined visual features; the text semantic features and the local visual features are then processed together. The visual features are input into the adaptive feature decoupling module for alignment and decoupling processing to obtain the mutual information loss value of the attribute-object combination in the training image; the mutual information loss value represents the degree of information sharing between attributes and objects; in the loss function module, based on the similarity between the corresponding text semantic features and the local visual features, the preset cross-entropy loss function and the preset mutual information loss function, the total loss function is determined, and the learning parameters in the text feature entanglement module, the weight parameters in the adaptive feature decoupling module and the combined word vector embedding are updated according to the backpropagation mechanism of the total loss function. The above training process is repeated until the total loss function converges to obtain the trained image recognition prediction model based on word vectors and the prediction recognition mutual information loss value of the attribute-object combination;

[0043] The model processing module is used to input the image to be classified into the trained image recognition model for recognition processing and to obtain the recognition result.

[0044] The above-described solution of the present invention has at least the following beneficial effects:

[0045] The image recognition method and apparatus provided by the above-described solution of the present invention, by determining an image recognition prediction model based on word vector guidance, wherein the image recognition prediction model includes: a text feature entanglement module, a visual feature representation module, an adaptive feature decoupling module, and a loss function module; the text word vector embeddings of training images in the training set are input into the text feature entanglement module, and the text word vector embeddings are interactively fused based on a preset guidance mechanism to obtain interactively fused text semantic features; the text word vector embeddings include attribute word vector embeddings, object word vector embeddings, and combined word vector embeddings, and the text semantic features include attribute semantic features, object semantic features, and combined semantic features; the global visual information corresponding to the training images in the training set is input into the visual feature representation module, and the global visual information is decoupled based on a preset attention mechanism to obtain local visual features in the training images, wherein the local visual features include attribute visual features, object visual features, and combined visual features; the text semantic features and the... The local visual features are input into the adaptive feature decoupling module for alignment and decoupling processing to obtain the mutual information loss value of the attribute-object combination in the training image. The mutual information loss value represents the degree of information sharing between attributes and objects. In the loss function module, based on the similarity between the corresponding text semantic features and the local visual features, the preset cross-entropy loss function, and the preset mutual information loss function, the total loss function is determined. The learning parameters in the text feature entanglement module, the weight parameters in the adaptive feature decoupling module, and the combined word vector embedding are updated according to the backpropagation mechanism of the total loss function. The above training process is repeated until the total loss function converges to obtain the trained image recognition prediction model based on word vectors and the predicted recognition mutual information loss value of the attribute-object combination. The image to be classified is input into the trained image recognition model for recognition processing to obtain the recognition result. This overcomes the problems of high entanglement in the visual feature space and semantic ambiguity, thereby improving the accuracy and efficiency of image recognition. Attached Figure Description

[0046] Figure 1 This is a flowchart of an image recognition method provided by an embodiment of the present invention;

[0047] Figure 2 This is an architecture diagram of an image recognition prediction model provided in an optional embodiment of the present invention;

[0048] Figure 3 This is a schematic block diagram of the module block of the image recognition device provided in the embodiments of the present invention;

[0049] Figure 4 This is a schematic block diagram of an electronic device provided in an embodiment of the present invention;

[0050] Figure 5 This is a schematic block diagram of a computing device provided in an embodiment of the present invention. Detailed Implementation

[0051] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0052] In the following description, certain specific details are set forth for the purpose of illustrating various disclosed embodiments in order to provide a thorough understanding of the various disclosed embodiments. However, those skilled in the art will recognize that embodiments may be practiced without one or more of these specific details. In other instances, well-known apparatuses, structures, and techniques associated with this application may not have been shown or described in detail to avoid unnecessarily obscuring the description of the embodiments.

[0053] Throughout this specification, references to "an embodiment" or "an embodiment" indicate that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Therefore, the appearance of "in an embodiment" or "an embodiment" in various places throughout the specification does not necessarily refer to the same embodiment. Furthermore, a particular feature, structure, or characteristic may be combined in any manner in one or more embodiments.

[0054] In the following description, in order to clearly demonstrate the structure and working method of the present invention, a number of directional terms will be used. However, terms such as "front", "back", "left", "right", "outside", "inside", "outward", "inward", "up", and "down" should be understood as convenient terms and not as limiting terms.

[0055] like Figure 1 As shown, an embodiment of the present invention provides an image recognition method, including:

[0056] Step 11: Determine the image recognition prediction model based on word vector guidance. The image recognition prediction model includes: a text feature entanglement module, a visual feature representation module, an adaptive feature decoupling module, and a loss function module.

[0057] Step 12: Embed the text word vectors of the training images in the training set into the input text feature entanglement module, and perform interactive fusion processing on the text word vector embedding based on the preset guidance mechanism to obtain the interactively fused text semantic features; the text word vector embedding includes attribute word vector embedding, object word vector embedding and combined word vector embedding, and the text semantic features include attribute semantic features, object semantic features and combined semantic features;

[0058] Step 13: Input the global visual information corresponding to the training images in the training set into the visual feature representation module, and decouple the global visual information based on the preset attention mechanism to obtain the local visual features in the training images. The local visual features include attribute visual features, object visual features and combined visual features.

[0059] Step 14: Input the text semantic features and local visual features into the adaptive feature decoupling module for alignment and decoupling processing to obtain the recognition mutual information loss value of the attribute-object combination in the training image; the recognition mutual information loss value represents the degree of information sharing between attributes and objects;

[0060] Step 15: In the loss function module, based on the similarity between the corresponding text semantic features and local visual features, the preset cross-entropy loss function and the preset mutual information loss function, the total loss function is determined, and the learning parameters in the text feature entanglement module, the weight parameters in the adaptive feature decoupling module and the combined word vector embedding are updated according to the backpropagation mechanism of the total loss function. The above training process is repeated until the total loss function converges, so as to obtain the trained image recognition prediction model based on word vector and the prediction recognition mutual information loss value of attribute-object combination.

[0061] Step 16: Input the image to be classified into the trained image recognition model for recognition processing and obtain the recognition result.

[0062] In this embodiment, the basic architecture of the image recognition prediction model is first determined, such as... Figure 2As shown, the image recognition prediction model includes a text feature entanglement module GER (Guided Embedding Refine), a visual feature representation module FRB (Focus Res Bloc), an adaptive feature decoupling module KDN (Kernelized Dependency Network), and a loss function module DACL (Decoupling-Alignment-Composition Loss Function). Specifically, the text feature entanglement module GER adaptively adjusts the text word vector embeddings through a pre-defined guidance mechanism to address semantic ambiguity; the visual feature representation module FRB primarily achieves initial decoupling of entangled visual features in training images through a pre-defined attention mechanism; the adaptive feature decoupling module KDN performs mutual information comparison learning, aligning different modal features while achieving deeper attribute-object decoupling at the embedding dimension level; finally, the loss function module DACL balances object and attribute classification losses, alignment losses, and combined classification losses, achieving synergistic optimization of feature decoupling, modality alignment, and combined prediction.

[0063] Here, the training set may include multiple training images and the attributes and objects corresponding to the training images; here, the attributes and objects of the training images are defined as: attribute sets. and collection of objects Then the set of all possible combinations of attribute-object can be represented as For example, red tomatoes ,red ,tomato The model needs to evaluate all combinations during training, including known combinations. s and unknown combinations u Among them, attributes are used in the model training phase. Object ,and s For training, s ∪ u Used for testing, where, s ∩ .

[0064] During model training, the contextual semantics of objects and attributes in the training images can first be refined. To address the semantic ambiguity of combined concepts, a pre-defined guidance mechanism in the text feature entanglement module dynamically interacts and adaptively fuses the text word vector embeddings corresponding to attributes and objects, making the text word vector embeddings more specific and accurate. Preferably, this pre-defined guidance mechanism can be a bidirectional guidance mechanism to guide the text word vector embeddings corresponding to attributes and objects separately. Secondly, the visual features corresponding to attributes and objects in the training images can be decoupled. To address the entanglement and ambiguity of visual features, this application can extract local visual features of attributes and objects in the training images separately through a pre-defined attention mechanism. This allows for the initial separation of the two representations in the early stages of visual encoding, improving the discriminability of subsequent features. Furthermore, in terms of global cross-modal alignment and decoupling, to address the problem of visual attribute ambiguity, the text semantic features and local visual features extracted in the above steps are globally aligned and decoupled in the adaptive feature decoupling module. Here, the nonlinear correlation between local visual features and text semantic features can be captured by comparative learning between positive and negative sample pairs composed of text semantic features and local visual features, and represented by the mutual information loss value. This achieves the alignment of corresponding attributes and objects and the decoupling of spurious combinations, preventing spurious associations and realizing the separation of attribute and object representations.

[0065] Furthermore, to collaboratively optimize the interactive fusion of text word vector embeddings, the decoupling of local visual features, and the alignment of text semantic features and local visual features, the loss function module incorporates a preset cross-entropy loss function based on the similarity between text semantic features and local visual features, and a preset mutual information loss function based on the mutual information loss value of recognition, to obtain the total loss function DACL. In the loss function module, by integrating all loss terms, semantic alignment between different modalities and concepts is promoted, so that while the model accurately identifies, it forces the model to learn meaningful dependencies and alignments between different modalities and concepts, thereby improving the model's generalization ability.

[0066] In the aforementioned iterative training process, the input training image, along with its corresponding attributes, objects, and combinations, are propagated forward through the text feature entanglement module, visual feature representation module, and adaptive feature decoupling module to obtain the corresponding prediction results. The difference between the prediction results and the true results is quantified using the total loss function, and the parameters involved in the text feature entanglement module, visual feature representation module, and adaptive feature decoupling module are updated based on gradient backpropagation. Simultaneously, the process of forward training propagation, loss calculation, gradient backpropagation, and parameter update is repeated until the total loss function converges, thus obtaining the trained image recognition prediction model based on word vector guidance.

[0067] In an optional embodiment of the present invention, the text word vector embedding of the training image is obtained by encoding the attribute text, object text and combined text in the training image based on a pre-trained word embedding encoder.

[0068] Before processing through the text feature entanglement module, a pre-trained word embedding encoder can be loaded first. The attribute word vectors and object word vectors are represented as word embedding features of attribute words, object words, and corresponding combined words. The above process can be described as follows:

[0069] ;

[0070] ;

[0071] in, Indicating object words, Indicates attribute words, and These represent the pre-trained word embedding encoders. The resulting object word vectors and attribute word vectors;

[0072] Further through a pre-trained word embedding encoder Extracting the internal and inter-textual relationships between object words and attribute words enhances the representational power of word embeddings. Furthermore, through a learnable embedding layer, combined word vectors are obtained. The above process can be represented as an embedded representation:

[0073] ;

[0074] ;

[0075] ;

[0076] Represents combined word vectors. Represents the word vector embedding of objects. How to represent attribute word vector embedding? Represents combined word vector embedding, As a learnable embedding layer, it can adaptively update the combined word vectors during model iterative training. The vector representation of , Representing the The attribute and object index in the combined pair.

[0077] In an optional embodiment of the present invention, step 12 above may include:

[0078] Step 121: Based on the preset attribute-guided embedding mechanism, the attribute word vector embeddings are weighted and fused into the object word vector embeddings to obtain the object semantic features;

[0079] Step 122: Based on the preset object-guided embedding mechanism, the object word vector embedding is weighted and fused into the attribute word vector embedding to obtain the attribute semantic features;

[0080] Step 123: Perform association processing on object semantic features, attribute semantic features, and combined word vector embeddings to obtain combined semantic features.

[0081] In this embodiment, the text feature entanglement module GER includes two core components: a preset attribute-guided embedding mechanism AGOER and a preset object-guided embedding mechanism OGAER. The preset attribute-guided embedding mechanism AGOER integrates attribute information into the object's visual features to enhance the attribute discriminative power of the object features; the preset object-guided embedding mechanism OGAER integrates object information into the attribute visual features to improve the object correlation of the attribute features.

[0082] Since the semantics of an object can be guided and generalized by its commonly used attributes (e.g., "table" can be paired with "wooden" and "round"), by calculating the attention between attribute word embeddings and object word embeddings, and then weightedly fusing the attribute word embeddings back into the object word embeddings, the object word embeddings can retain the semantic features of relevant attributes, thereby enhancing their ability to recognize unseen combinations. The above process can be represented as:

[0083] ;

[0084] ;

[0085] in, This represents the enhanced semantic features of an object after incorporating attribute word vector embeddings. It is attribute word vector embedding The attribute transpose matrix, where γ is a learnable parameter to control the strength of cross-modal information fusion;

[0086] Similarly, by focusing attribute word embeddings on object word embeddings, attribute embeddings can better generalize to the context of new object words. For example, the attribute "circle" can be represented by objects it commonly modifies, such as "ball" and "table." The above process can be represented as follows:

[0087] ;

[0088] ;

[0089] in, This represents enhanced attribute semantic features that incorporate object word vector embeddings. It is object word vector embedding The object transpose matrix;

[0090] Furthermore, attribute semantic features Object semantic features Combined word vectors This constitutes a combined semantic feature that comprehensively includes all relevant semantic information. The above process can be represented as:

[0091] ;

[0092] In the text feature entanglement module, a cross-attention mechanism is introduced: namely, the pre-defined attribute-guided embedding mechanism AGOER and the pre-defined object-guided embedding mechanism OGAER. This promotes information exchange and fusion between attribute and object features, effectively solving the problem of limited generalization ability caused by simply using attribute-object pairs in traditional methods. It makes full use of the inherent contextual dependencies between attributes and objects, enabling the model to adapt to semantic drift caused by different combinations, thereby significantly enhancing the model's ability to recognize unseen combinations.

[0093] In an optional embodiment of the present invention, step 13 above may include:

[0094] Step 131: Map the input training image based on the preset visual encoder to obtain the global visual information of the training image.

[0095] Step 132: Extract features from global visual information based on a preset attention mechanism to obtain local visual features.

[0096] In this embodiment, the preset visual encoder can be a Wave-MLP visual encoder to capture global visual information of the training images;

[0097] Furthermore, a preset attention mechanism is used to decouple global visual information to extract corresponding attributes, objects, and their combinations of local visual features. Preferably, the preset attention mechanism can include a preset spatial attention mechanism (SFRB) and a preset channel attention mechanism (CFRB). The SFRB adaptively focuses on visual regions that can characterize the shape and position of the core object, while the CFRB adaptively focuses on visual cues that can capture the diversity of object attributes. The SFRB employs a spatial attention mechanism.

[0098] Visual feature extraction is performed through a separate attention mechanism in the visual feature representation module, which enhances the model's discriminative power and helps to initially decouple the mixed attribute and object representations in visual features at the visual level, thus alleviating the problem of mixed attribute and object features.

[0099] In an optional embodiment of the present invention, step 132 above may include:

[0100] Step 1321: Assign global visual information to object visual information and attribute visual information, and obtain combined visual information based on object visual information and attribute visual information;

[0101] Step 1322: Extract features from attribute visual information based on a preset spatial attention mechanism to obtain attribute visual features;

[0102] Step 1323: Extract features from the object's visual information based on a preset channel attention mechanism to obtain the object's visual features;

[0103] Step 1324: Extract features from the combined visual information based on a preset multi-layer perception mechanism to obtain combined visual features.

[0104] In this embodiment, the global visual information is first obtained by mapping the input training image M through a visual encoder. Assigning global visual information to object visual information and attribute visual information can be specifically represented as follows:

[0105] ;

[0106] ;

[0107] in, Visual information representing attributes, Represents the visual information of an object;

[0108] Furthermore, based on object visual information and attribute visual information, the average value of all spatial locations in each channel is taken to obtain combined visual information. Specifically, it can be expressed as:

[0109] ;

[0110] Flatten(.) represents flattening a multidimensional tensor into a one-dimensional vector, which helps to capture the comprehensive semantics of the entire training image.

[0111] Furthermore, based on a pre-defined spatial attention mechanism Feature extraction of attribute visual information allows the model to focus on spatial regions in the image features that are more relevant to object recognition, thus obtaining attribute visual features. The specific process can be represented as follows:

[0112] ;

[0113] ;

[0114] Among them, the pre-set spatial attention mechanism This allows the model to focus on spatial location and shape information related to attribute visual information, thus obtaining initial attribute visual features. , This is an attribute-based multilayer perceptron, designed to project the visual features of attributes onto a shared embedding space to obtain attribute visual features. .

[0115] Preferably, a pre-defined spatial attention mechanism is used. The specific structure is as follows:

[0116] ;

[0117] ;

[0118] in, Indicates intermediate parameters. Indicates a convolutional layer. Indicates the batch normalization layer. This represents the activation function. This represents element-wise multiplication, which applies the attention weight map element-wise to the feature map, thereby weighting or enhancing the features. and These represent the average pooling and max pooling methods, respectively. This represents intermediate parameters, specifically as follows:

[0119] .

[0120] Furthermore, based on the preset channel attention mechanism CFRB, feature extraction is performed on the visual information of the object, enhancing the channel information in the image features that is more important for attribute recognition, thereby obtaining the visual features of the object. The specific process can be represented as follows:

[0121] ;

[0122] ;

[0123] in, Represents the initial visual features of the object associated with it. This is an object multilayer perceptron, designed to project the visual features of objects onto a shared embedding space to obtain the object's visual features. .

[0124] Preferably, the specific structure of the preset channel attention mechanism CFRB is as follows:

[0125] ;

[0126] ;

[0127] ;

[0128] ;

[0129] ;

[0130] in, , Both represent intermediate parameters.

[0131] Furthermore, by pre-setting a multi-layered perception mechanism Feature extraction is performed on the combined visual information to obtain combined visual features that represent the entire image context. Combining visual features Specifically, it can be expressed as:

[0132] .

[0133] Visual feature extraction using a separate attention mechanism helps to initially decouple the mixed attributes and object representations in visual features at the visual level, alleviating the problem of mixed attributes and object features.

[0134] In an optional embodiment of the present invention, step 14 above may include:

[0135] Step 141: Pair the text semantic features and local visual features to form the input sample pairs for the adaptive feature decoupling module. The input sample pairs include text semantic feature sample pairs and local visual feature sample pairs.

[0136] In this embodiment, multiple textual semantic features and multiple local visual features are randomly paired, and the paired sample pairs are used as input sample pairs for the adaptive feature decoupling module to form input sample pairs for mutual information estimation. , where x and y can be textual semantic features or local visual features, and are both samples in an input sample pair.

[0137] Furthermore, step 141 above may include:

[0138] Step 1411: Based on the preset semantic relationship, the text semantic features and local visual features are grouped to obtain positive sample pairs and negative sample pairs. The positive sample pairs contain multiple text semantic feature sample pairs with a first correlation relationship and multiple local visual feature sample pairs with a second correlation relationship. The negative sample pairs contain multiple text semantic feature sample pairs with a third correlation relationship and multiple local visual feature sample pairs with a fourth correlation relationship. The first correlation relationship is greater than the third correlation relationship, and the second correlation relationship is greater than the fourth correlation relationship.

[0139] Here, the presupposed semantic relationship is the strength of association between different textual semantic features and local visual features, which can be represented by the association relationship. For example, for an image of a wet cat, the relationship between the attribute visual feature "wet" and the object visual feature "cat" is not strong, but the relationship between the attribute visual feature "wet" and the combined visual feature "wet cat" is strong. That is, the second association relationship between the local visual feature sample pair: attribute visual feature "wet" and combined visual feature "wet cat" is greater than the second association relationship between the local visual feature sample pair: attribute visual feature "wet" and object visual feature "cat".

[0140] Based on predefined semantic relationships, textual semantic features and local visual features are grouped separately to obtain positive and negative sample pairs, which can be specifically represented as follows:

[0141] ;

[0142] ;

[0143] in, Indicates a positive sample pair; Indicates negative sample pairs; Dimensional transpose representing the visual features of an object; This represents the dimensional transpose of the visual features of the attributes. By constructing positive and negative sample pairs, positive sample pairs promote modality alignment and knowledge transfer, thereby achieving modality alignment and effective knowledge transfer. Simultaneously, by further decoupling the negative sample pairs, spurious associations are prevented, thus preventing the model from learning false relevance. Furthermore, the dimensionality of the object's visual feature attributes is transposed from (… ) converted to ( This allows the model to decouple images from different embedding dimensions.

[0144] In an optional embodiment of the present invention, step 14 may further include:

[0145] Step 142: The adaptive feature decoupling module performs concatenation mapping on the samples in the input sample pairs to obtain the joint sample distribution and marginal sample distribution of the input sample pairs; the joint sample distribution is generated by positive sample pairs, and the input sample pairs are obtained from the sampling of positive sample pairs. In this context, there are semantic relationships between them. The marginal sample distribution represents the combination method through random sampling, allowing the sampled samples to be combined. and samples They are mutually independent, which is equivalent to being independent of each other. and samples Independent sampling allows the constructed negative sample pairs to eliminate the spurious correlation between X and Y sample pairs.

[0146] Here, any input sample pair is obtained by sampling positive or negative sample pairs. The specific process of performing splicing and mapping on the samples in the dataset is as follows:

[0147] ;

[0148] ;

[0149] Among them, the sample and samples The corresponding samples are concatenated to form a joint sample set. ; take the sample Random samples are obtained by randomly shuffling the samples in the sample. Then the sample and random samples The corresponding samples are spliced ​​together to form the edge sample set. ;here, The mapping multilayer perceptron aims to learn feature representations that are sensitive to dependencies. The joint samples and edge samples are mapped into the hidden space respectively to obtain their joint sample distribution in the learned feature space. and marginal sample distribution .

[0150] In an optional embodiment of the present invention, step 14 may further include:

[0151] Step 143: Determine the mutual information loss value of attribute-object combination recognition in the training image based on the joint sample distribution and the edge sample distribution.

[0152] Specifically, step 143 above may include:

[0153] Step 1431: Determine the Gaussian kernel value between sample pairs in the joint sample distribution and the marginal sample distribution based on the preset Gaussian kernel function, so as to quantify the difference between sample pairs in the joint sample distribution and the marginal sample distribution;

[0154] Step 1432: Based on the Gaussian kernel value and the preset loss weight of positive sample pairs, determine the first recognition mutual information loss value of the attribute-object combination in the positive sample pair;

[0155] Step 1433: Based on the Gaussian kernel value and the preset loss weight of negative sample pairs, determine the second recognition mutual information loss value of the attribute-object combination in the negative sample pair.

[0156] In this embodiment, the joint sample distribution is quantified by using a preset Gaussian kernel function MMD. With marginal sample distribution Differences between products and calculate the joint sample distribution. With marginal sample distribution The Gaussian kernel values ​​between all sample pairs are used to construct the kernel matrix and decompose it. Specifically, it can be represented as:

[0157] ;

[0158] ;

[0159] in, , The expected values ​​represent the joint sample distributions, respectively. With marginal sample distribution The average kernel value among all sample pairs extracted from the distribution is used to measure the average similarity between samples within the distribution. Represents a sampling operation. , Indicates from the joint sample distribution Independently sampled samples , Indicates the distribution of marginal samples Independently sampled samples, Gaussian kernel function (that is, in the formula) , , This measures the similarity between two samples in a high-dimensional feature space. It is an adaptive parameter, representing the bandwidth value. This indicates the number of Gaussian kernels with different bandwidths. The expected value is represented from the joint sample distribution. With marginal sample distribution A sample is drawn from each of the two distributions, and the average kernel value between the two samples is calculated to reflect the average similarity between samples from different distributions.

[0160] By using Gaussian kernels with multiple bandwidths, the pre-defined Gaussian kernel function MMD can more sensitively capture dependencies at different scales. Finally, based on the pre-defined Gaussian kernel function MMD, the statistical dependency between positive and negative sample pairs is quantified, thereby obtaining an optimizable mutual information surrogate term (i.e., the recognition mutual information loss value corresponding to the positive and negative sample pairs respectively), which can be specifically expressed as:

[0161] ;

[0162] ;

[0163] in, This represents the first mutual information loss value for positive sample pairs; This represents the second identification mutual information loss value for negative sample pairs; This indicates the preset loss weights for positive sample pairs; This indicates the preset loss weights for negative sample pairs.

[0164] For positive sample pairs, maximizing the first recognition mutual information loss value, making the difference between the product of their joint distribution and marginal distribution as large as possible, allows for sufficient information sharing between representations of different modalities and different concepts (such as attributes and objects), thereby achieving modality alignment and effective knowledge transfer. For negative sample pairs, minimizing the second recognition mutual information loss value, making the difference between the product of their joint distribution and marginal distribution as small as possible, ensures that the model can distinguish irrelevant concepts or modalities.

[0165] In this embodiment, the adaptive feature decoupling module starts from the embedding dimension and performs "lateral" sampling on the category embedding of each sample in the input sample pair. Each independent embedding dimension is treated as a "sample," guiding the model to learn decoupled attributes and object semantics in these dimensions. For example, if the k-th dimension is learned to have a high response to the "red" attribute, then this dimension will be strengthened to produce similarly high responses to all combined concepts containing "red" (such as "red car" and "red ball"). In this way, the adaptive feature decoupling module achieves "natural" decoupling of attributes and objects at the category embedding level (e.g., "red car" is decomposed into the independent semantics of "red" and "car"). This dual decoupling of instance and embedding dimensions allows abstract category concepts to be better aligned in a shared embedding space and presents a highly structured organization, promoting the model's understanding and reasoning ability for unseen combined concepts.

[0166] In an optional embodiment of the present invention, the total loss function is expressed as:

[0167] ;

[0168] in, , , Both represent the preset cross-entropy loss, used to measure the difference between the model's prediction and the true label; This represents the preset mutual information loss function obtained based on the mutual information loss value of attribute-object combination. , The first weight used to balance the attribute-object combined loss with the independent attribute or object loss. The second weight represents the loss of an independent attribute or object. This represents a regularization term used to balance the contribution of the mutual information loss value of attribute-object combinations to the total loss.

[0169] In this embodiment, to achieve the optimal balance between modality combination and alignment, a total loss function is obtained by incorporating mutual information loss. By integrating all loss terms, semantic alignment between different modalities and concepts is promoted, enabling the model to learn meaningful dependencies and alignments between different modalities and concepts while accurately classifying them, thereby improving the model's generalization ability. The calculation process of the predicted score in the loss function module is as follows:

[0170] ;

[0171] ;

[0172] ;

[0173] Specifically, the similarity between visual features and corresponding semantic features is calculated using dot product to obtain prediction scores for objects, attributes, and combination pairs, respectively. , , At the same time, a preset cross-entropy loss function is introduced. , , To supervise model training, here, This represents the preset cross-entropy loss function corresponding to the object. This represents the preset cross-entropy loss function corresponding to the attribute. This represents the preset cross-entropy loss function corresponding to the combination pair; each preset cross-entropy loss function can be specifically expressed as:

[0174] ;

[0175] ;

[0176] ;

[0177] in, , , Representing the The index of the true category labels for each sample, corresponding to the object, attribute, and combination pair. , , They are the first The predicted probability distribution of each sample output by the model, where B represents the batch dimension. By supervising the integrated modal information through these pre-defined cross-entropy loss functions, the model can balance the importance of different tasks during the learning process, and finally guide the end-to-end learning of the model by calculating multi-task loss.

[0178] The following will evaluate the solution provided in the above embodiments using specific comparative data on different public datasets, MIT-States, C-GQA, and UT-Zappos50k, in conjunction with the methods provided in the above embodiments and other different existing algorithms. The details are as follows:

[0179] The data sets were segmented and statistically analyzed, as shown in Table 1. In the table, A represents an attribute, O represents an object, C represents a combination, s represents a known combination, u represents an unseen combination, and I represents an image.

[0180] Table 1. Dataset Segmentation Statistics Table

[0181]

[0182] For word embedding initialization, word2vec was used as the pre-trained word vectors on the C-GQA dataset, while GloVe word vectors were used on the UT-Zappos50K and MIT-States datasets. The word embedding dimension was set to 300 for all datasets. Visual feature extraction adopted a Wave-MLP structure, and its output feature dimension was 512.

[0183] In the visual feature projection section, the visual features of attributes, objects, and combination pairs are respectively projected through... , , Projection is performed, and each MLP contains two fully connected layers. Except for the object branch, the output dimension of each layer is [600, 512], and the output dimension of the object branch is [600, 1024]. Attribute and object word embeddings are used. , The encoder layers were set to 4 layers for UT-Zappos50K and 4 layers for MIT-States, respectively. CGQA had no encoder layers. During training, the sample size per batch was set to 128, and during testing, it was set to 32. The maximum number of training epochs was set to 50. The optimizer used was Adam, the principal feature dimension of the model was 512, the regularization term was set to 0.05, and the initial learning rate was 2 × 10⁻⁶.-5 Meanwhile, the total weight of the adaptive feature decoupling module loss is 0.1, where the weight of the loss for positive sample pairs is preset. Setting it to 0 presets the loss weights for negative samples. The weight parameter α is 0.3 when the attribute-object-combination score is fused, and the model reaches its optimal performance. At this time, the weight coefficient β of the combination loss and the attribute / object loss is 0.7.

[0184] Table 2 shows the prediction results of each method in closed scenarios such as CGQA, UTzappos50K, and MIT-States (where S, U, H, and AUC represent the Top-1 accuracy for each dataset). It can be seen that the method of this invention achieves significant improvements in multiple metrics, outperforming almost all benchmark models, and demonstrating a significant performance improvement. Specifically, for Utzappos50K, DREAM-CZSL improves the prediction accuracy S for known combinations, the prediction accuracy U for unseen combinations, and the area under the harmonic mean H and SU relationship curve AUC from suboptimal results of 63.5%, 67.5%, 51.3%, and 35.4% to 68.0%, 72.1%, 55.3%, and 42.8%, respectively, representing improvements of 4.45%, 4.64%, 4.01%, and 7.37%. This demonstrates the superior ability of the method of this invention in balancing the recognition of known and unseen combinations and effectively improves the overall generalization performance. For MIT-States, despite significant noise interference and large inter-class differences, the method of this invention still achieved state-of-the-art accuracies of 35.0%, 31.3%, 23.1%, and 8.0% on S, U, H, and AUC, respectively, representing improvements of 2.83%, 1.41%, 3.57%, and 2.03% compared to the suboptimal results. C-GQA is one of the most challenging datasets due to its large number of combinations.

[0185] As shown in Table 2, due to the decoupling of multi-stage features and the alignment of global visual and semantic features, the unseen combination reasoning ability of the method of the present invention is significantly improved, with an improvement of 5.33% compared to the suboptimal result. Although the accuracy of the method of the present invention in seen combinations is lower than that of DBC, the method provided in this application improves the H and AUC by 1.3% and 0.9%, respectively. This further demonstrates that when it is difficult to distinguish individual components, the method of the present invention can achieve a good balance in the identification of seen and unseen combinations, illustrating the effectiveness of the method of the present invention in handling complex combinations and generalization tasks.

[0186] Table 2. Comparison of validation data for different methods on UTZappos50K, CGQA, and MIT-States.

[0187]

[0188] The image recognition method and apparatus provided in the above embodiments of the present invention overcome the problems of high entanglement in visual feature space and semantic ambiguity by decoupling and aligning visual features and semantic features in a multi-stage process. Specifically, by determining an image recognition prediction model based on word vector guidance, the image recognition prediction model includes: a text feature entanglement module, a visual feature representation module, an adaptive feature decoupling module, and a loss function module; text word vectors of training images in the training set are embedded into the text feature entanglement module, and interactive fusion processing is performed on the text word vector embedding based on a preset guidance mechanism to obtain interactively fused text semantic features; text word vector embedding includes attribute word vector embedding, object word vector embedding, and combined word vector embedding, and text semantic features include attribute semantic features, object semantic features, and combined semantic features; global visual information corresponding to the training images in the training set is input into the visual feature representation module, and decoupling processing is performed on the global visual information based on a preset attention mechanism to obtain local visual features in the training images, local visual features including attribute visual features, object visual features, and combined visual features; text semantic features and local visual features are input into the adaptive feature decoupling module for alignment. The system employs a decoupling process to obtain the mutual information loss value for attribute-object combinations in the training image. This mutual information loss value represents the degree of information sharing between attributes and objects. In the loss function module, based on the similarity between corresponding textual semantic features and local visual features, a preset cross-entropy loss function, and a preset mutual information loss function, the total loss function is determined. The learning parameters in the text feature entanglement module, the weight parameters in the adaptive feature decoupling module, and the combined word vector embedding are updated according to the backpropagation mechanism of the total loss function. This training process is repeated until the total loss function converges, resulting in a well-trained image recognition prediction model guided by word vectors and the predicted mutual information loss value for attribute-object combinations. The image to be classified is input into the trained image classification model for recognition processing, and the recognition result is obtained. This overcomes the problems of high entanglement in the visual feature space and semantic ambiguity, and improves the generalization ability of the image recognition prediction model to complex and diverse unknown combinations, thereby enhancing the accuracy and efficiency of image recognition.

[0189] like Figure 3 As shown, embodiments of the present invention also provide an image recognition device 30, comprising:

[0190] The model building module 31 is used to determine the image recognition prediction model based on word vector guidance. The image recognition prediction model includes: a text feature entanglement module, a visual feature representation module, an adaptive feature decoupling module, and a loss function module.

[0191] The model training module 32 is used to embed the text word vectors of the training images in the training set into the input text feature entanglement module, and to perform interactive fusion processing on the text word vector embedding based on a preset guidance mechanism to obtain the interactively fused text semantic features. The text word vector embedding includes attribute word vector embedding, object word vector embedding, and combined word vector embedding, and the text semantic features include attribute semantic features, object semantic features, and combined semantic features. The global visual information corresponding to the training images in the training set is input into the visual feature representation module, and the global visual information is decoupled based on a preset attention mechanism to obtain the local visual features in the training images. The local visual features include attribute visual features, object visual features, and combined visual features. The text semantic features and local visual features are then combined. The visual features are input into the adaptive feature decoupling module for alignment and decoupling processing to obtain the mutual information loss value of the attribute-object combination in the training image. The mutual information loss value represents the degree of information sharing between attributes and objects. In the loss function module, the total loss function is determined based on the similarity between the corresponding text semantic features and local visual features, the preset cross-entropy loss function, and the preset mutual information loss function. The learning parameters in the text feature entanglement module, the weight parameters in the adaptive feature decoupling module, and the combined word vector embedding are updated according to the backpropagation mechanism of the total loss function. The above training process is repeated until the total loss function converges to obtain the trained image recognition prediction model based on word vectors and the prediction recognition mutual information loss value of the attribute-object combination.

[0192] The model processing module 33 is used to input the image to be classified into the trained image classification model for recognition processing and to obtain the recognition result.

[0193] It should be noted that this device is a device corresponding to the above-described image recognition method. All implementation methods in the above method embodiments are applicable to this embodiment and can achieve the same technical effect.

[0194] like Figure 4As shown, embodiments of the present invention also provide an electronic device 50, including: a memory 51 for storing one or more computer programs; and one or more processors 52 for executing the one or more computer programs. When the computer programs are run by the processors, they perform the image recognition method as described above. All implementations in the above method embodiments are applicable to this embodiment and can achieve the same technical effects. The electronic device 50 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components, their connections and relationships, and their functions shown in this invention are merely examples and are not intended to limit the implementation of the invention described and / or claimed herein.

[0195] like Figure 5 As shown, electronic device 50 is a computing device or computer system, which may include CPU 501 (computing unit), which can perform various appropriate actions and processes according to a computer program stored in ROM 502 (read-only memory) or a computer program loaded from storage unit 508 into random access RAM 503 (memory). RAM 503 may also store various programs and data required for the operation of device 500. CPU 501, ROM 502, and RAM 503 are interconnected via bus 504. I / O interface 505 (input / output interface) is also connected to bus 504.

[0196] Multiple components in electronic device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0197] CPU 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of CPU 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. CPU 501 performs the various methods and processes described above. For example, in some embodiments, the image recognition method can be implemented as a computer software program tangibly contained in a computer-readable storage medium, such as storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by CPU 501, one or more steps of the image recognition method described above can be performed. Alternatively, in other embodiments, CPU 501 can be configured to perform the image recognition method by any other suitable means (e.g., by means of firmware).

[0198] Embodiments of the present invention also provide a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the image recognition method as described above. All implementations in the above method embodiments are applicable to this embodiment and can achieve the same technical effects.

[0199] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0200] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0201] In the embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0202] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0203] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0204] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0205] Furthermore, it should be noted that in the apparatus and method of the present invention, it is obvious that the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered equivalent solutions of the present invention. Moreover, the steps performing the above series of processes can naturally be executed in the order described, but are not necessarily required to be executed in chronological order; some steps can be executed in parallel or independently of each other. Those skilled in the art will understand that all or any step or component of the method and apparatus of the present invention can be implemented in any computing device (including processors, storage media, etc.) or network of computing devices, in hardware, firmware, software, or a combination thereof. This is something that those skilled in the art can achieve by using their basic programming skills after reading the description of the present invention.

[0206] Therefore, the object of the present invention can also be achieved by running a program or a set of programs on any computing device. The computing device can be a known general-purpose device. Therefore, the object of the present invention can also be achieved simply by providing a program product containing program code for implementing the method or apparatus. That is, such a program product also constitutes the present invention, and the storage medium storing such a program product also constitutes the present invention. Obviously, the storage medium can be any known storage medium or any storage medium developed in the future. It should also be noted that in the apparatus and method of the present invention, it is obvious that the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered equivalent to the present invention. Furthermore, the steps for performing the above series of processes can naturally be performed in the order described, but are not necessarily required to be performed in chronological order. Some steps can be performed in parallel or independently of each other.

[0207] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An image recognition method, characterized in that, include: A word vector-guided image recognition prediction model is determined, which includes: a text feature entanglement module, a visual feature representation module, an adaptive feature decoupling module, and a loss function module; The text word vectors of the training images in the training set are embedded into the text feature entanglement module, and interactive fusion processing is performed on the text word vector embeddings based on a preset guidance mechanism to obtain the interactively fused text semantic features; the text word vector embeddings include attribute word vector embeddings, object word vector embeddings, and combined word vector embeddings, and the text semantic features include attribute semantic features, object semantic features, and combined semantic features; Based on a preset attribute-guided embedding mechanism, the attribute word vector embeddings are weighted and fused into the object word vector embeddings to obtain the semantic features of the object; Based on a preset object-guided embedding mechanism, the object word vector embeddings are weighted and fused into the attribute word vector embeddings to obtain the attribute semantic features; The global visual information corresponding to the training images in the training set is input into the visual feature representation module, and the global visual information is decoupled based on a preset attention mechanism to obtain local visual features in the training images. The local visual features include attribute visual features, object visual features, and combined visual features. The global visual information is assigned to the object visual information and the attribute visual information, and the attribute visual information is used to extract features based on a preset spatial attention mechanism to obtain the attribute visual features. The visual information of the object is extracted based on a preset channel attention mechanism to obtain the visual features of the object; The text semantic features and the local visual features are input into the adaptive feature decoupling module for alignment and decoupling processing to obtain the mutual information loss value of the attribute-object combination in the training image; the mutual information loss value represents the degree of information sharing between attributes and objects. In the loss function module, based on the similarity between the corresponding text semantic features and the local visual features, the preset cross-entropy loss function and the preset mutual information loss function, the total loss function is determined, and the learning parameters in the text feature entanglement module, the weight parameters in the adaptive feature decoupling module and the combined word vector embedding are updated according to the backpropagation mechanism of the total loss function. The above training process is repeated until the total loss function converges, so as to obtain the trained image recognition prediction model based on word vector and the prediction recognition mutual information loss value of attribute-object combination. The image to be classified is input into a trained image recognition model for recognition processing, and the recognition result is obtained.

2. The image recognition method according to claim 1, characterized in that, The text word vector embedding of the training image is obtained by encoding the attribute text, object text, and combined text in the training image based on a pre-trained word embedding encoder.

3. The image recognition method according to claim 1, characterized in that, The text word vectors of the training images in the training set are embedded into the text feature entanglement module, and interactive fusion processing is performed on the text word vector embeddings based on a preset guidance mechanism to obtain the interactively fused text semantic features, including: The semantic features of the object, the semantic features of the attribute, and the combined word vector embedding are correlated to obtain the combined semantic features.

4. The image recognition method according to claim 1, characterized in that, The global visual information corresponding to the training images in the training set is input into the visual feature representation module, and the global visual information is decoupled based on a preset attention mechanism to obtain local visual features in the training images, including: The input training image is mapped based on a preset visual encoder to obtain the global visual information of the training image. Based on the preset attention mechanism, feature extraction is performed on the global visual information to obtain the local visual features.

5. The image recognition method according to claim 4, characterized in that, Based on the preset attention mechanism, feature extraction is performed on the global visual information to obtain the local visual features, including: The global visual information is assigned to the object visual information and the attribute visual information, and combined visual information is obtained based on the object visual information and the attribute visual information. The combined visual information is feature extracted based on a preset multi-layer perception mechanism to obtain the combined visual features.

6. The image recognition method according to claim 1, characterized in that, The text semantic features and the local visual features are input into the adaptive feature decoupling module for alignment and decoupling processing to obtain the mutual information loss value of the attribute-object combination in the training image, including: The text semantic features and the local visual features are paired to form the input sample pair of the adaptive feature decoupling module. The input sample pair includes text semantic feature sample pairs and local visual feature sample pairs. The adaptive feature decoupling module performs concatenation mapping on the samples in the input sample pair to obtain the joint sample distribution and marginal sample distribution of the input sample pair; Based on the joint sample distribution and the edge sample distribution, the mutual information loss value of the attribute-object combination in the training image is determined.

7. The image recognition method according to claim 6, characterized in that, Also includes: Based on preset semantic relationships, the text semantic features and the local visual features are grouped to obtain positive sample pairs and negative sample pairs. The positive sample pairs contain multiple text semantic feature sample pairs with a first correlation relationship and multiple local visual feature sample pairs with a second correlation relationship. The negative sample pairs contain multiple text semantic feature sample pairs with a third correlation relationship and multiple local visual feature sample pairs with a fourth correlation relationship. The first correlation relationship is greater than the third correlation relationship, and the second correlation relationship is greater than the fourth correlation relationship.

8. The image recognition method according to claim 7, characterized in that, Based on the joint sample distribution and the edge sample distribution, the mutual information loss value of the attribute-object combination in the training image is determined, including: The Gaussian kernel value between sample pairs in the joint sample distribution and the marginal sample distribution is determined based on a preset Gaussian kernel function, so as to quantify the difference between sample pairs in the joint sample distribution and the marginal sample distribution; Based on the Gaussian kernel value and the preset loss weight for positive sample pairs, the first mutual information loss value of the attribute-object combination in the positive sample pair is determined; Based on the Gaussian kernel value and the preset negative sample pair loss weight, the second recognition mutual information loss value of the attribute-object combination in the negative sample pair is determined.

9. The image recognition method according to claim 6, characterized in that, The total loss function is expressed as: ； in, , , Both are represented by the preset cross-entropy loss, which is used to measure the difference between the model prediction and the true label; This represents the preset mutual information loss function obtained based on the mutual information loss value of attribute-object combination. , The first weight used to balance the attribute-object combined loss with the independent attribute or object loss. The second weight represents the loss of an independent attribute or object. This represents a regularization term used to balance the contribution of the mutual information loss value of attribute-object combinations to the total loss.

10. An image recognition device, characterized in that, include: The model building module is used to determine the image recognition prediction model based on word vector guidance. The image recognition prediction model includes: a text feature entanglement module, a visual feature representation module, an adaptive feature decoupling module, and a loss function module. The model training module is used to input the text word vector embeddings of training images in the training set into the text feature entanglement module, and perform interactive fusion processing on the text word vector embeddings based on a preset guidance mechanism to obtain interactively fused text semantic features; the text word vector embeddings include attribute word vector embeddings, object word vector embeddings, and combined word vector embeddings, and the text semantic features include attribute semantic features, object semantic features, and combined semantic features; global visual information corresponding to the training images in the training set is input into the visual feature representation module, and the global visual information is decoupled based on a preset attention mechanism to obtain local visual features in the training images, the local visual features include attribute visual features, object visual features, and combined visual features; the text semantic features and the local visual features are then processed together. The visual features are input into the adaptive feature decoupling module for alignment and decoupling processing to obtain the mutual information loss value of the attribute-object combination in the training image; the mutual information loss value represents the degree of information sharing between attributes and objects; in the loss function module, based on the similarity between the corresponding text semantic features and the local visual features, the preset cross-entropy loss function and the preset mutual information loss function, the total loss function is determined, and the learning parameters in the text feature entanglement module, the weight parameters in the adaptive feature decoupling module and the combined word vector embedding are updated according to the backpropagation mechanism of the total loss function. The above training process is repeated until the total loss function converges to obtain the trained image recognition prediction model based on word vectors and the prediction recognition mutual information loss value of the attribute-object combination; The model processing module is used to input the image to be classified into the trained image recognition model for recognition processing and to obtain the recognition result.

Images