Cross-modal semantic alignment method, device and equipment based on fine-grained semantic decoupling

Abstract

The application provides a cross-modal semantic alignment method, device and equipment based on fine-grained semantic decoupling, which comprises the following steps: inputting an image to be aligned into a pre-training image encoder of a cross-modal semantic alignment model to obtain global semantic features; using a semantic feature extraction module in a high-layer coding block of the image encoder and performing high-layer visual feature extraction based on a preset query vector to obtain visual fine-grained semantic features, and inputting the visual fine-grained semantic features into a semantic feature decoupling module to obtain fine-grained semantic features of attributes, objects and combinations; inputting prompt words corresponding to the attributes, objects and combinations in the image to be aligned into a text encoder to extract text features to obtain soft prompt text features of the attributes, objects and combinations; inputting the global semantic features, the fine-grained semantic features corresponding to the attributes, objects and combinations and the soft prompt text features into a loss calculation module to perform image-text alignment, and performing linear processing on an alignment result to obtain an alignment prediction result.

Description

Technical Field

[0001] This invention relates to the field of cross-modal semantic alignment technology in computer vision, and in particular to cross-modal semantic alignment methods, apparatus and equipment based on fine-grained semantic decoupling. Background Technology

[0002] Current mainstream combinatorial zero-shot learning methods have evolved from a model that uses pre-trained image encoders and pre-trained word embedding encoders for image-text feature alignment to a new paradigm that uses large-scale pre-trained visual-language models (such as CLIP) as the backbone architecture. The latter loads learnable attributes and object embeddings into cue word templates (e.g., "a photo of [attribute] [object]"), then projects them into the text space using a pre-trained text encoder, performing cross-modal alignment with image features obtained from a pre-trained image encoder. Due to its strong prior knowledge, it has achieved good results in CZSL (Compositional Zero-Shot Learning) tasks. However, this CLIP-based model framework also introduces new challenges.

[0003] 1) Insufficient fine-grained semantic awareness. CLIP encodes the entire image into a fixed-dimensional global feature vector through a pre-trained image encoder and aligns it with text through contrastive learning. The model focuses more on the consistency of overall semantics, which is insufficient for tasks like zero-order combination learning, which requires matching more fine-grained semantic concepts in the image. It is prone to misjudgment when faced with unseen combinations.

[0004] 2) Insufficient decoupling between attributes and objects. Existing CZSL methods typically utilize global visual features obtained from CLIP to explicitly model attributes and objects. However, due to the limited representational power of global visual features, attributes and objects cannot be fully decoupled. For example, when the model encounters the combination "striped shirt," it prioritizes aligning features with the overall semantic concept, weakening the fine-grained features of "stripes" and "shirt," making it difficult to accurately learn their respective semantic features. It is also prone to bias towards combinations that have appeared in the training set. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a cross-modal semantic alignment method, apparatus and device based on fine-grained semantic decoupling, which improves the accuracy and efficiency of cross-modal semantic alignment by fully decoupling fine-grained semantics.

[0006] To address the aforementioned technical problems, embodiments of the present invention provide a cross-modal semantic alignment method based on fine-grained semantic decoupling, comprising:

[0007] Obtain the image to be aligned;

[0008] The image to be aligned is input into the pre-trained image encoder of the cross-modal semantic alignment model to obtain global semantic features. The semantic feature extraction module is used in the high-level coding block of the image encoder, and high-level visual features are extracted from the image to be aligned based on the preset query vector to obtain the visual fine-grained semantic features corresponding to the image to be aligned.

[0009] The visual fine-grained semantic features are input into the semantic feature decoupling module of the cross-modal semantic alignment model for decoupling processing to obtain attribute fine-grained semantic features, object fine-grained semantic features, and combined fine-grained semantic features.

[0010] The image to be aligned and the text labels corresponding to the attributes, objects and combinations in the image to be aligned are input into the text encoder of the cross-modal semantic alignment model, and text features are extracted based on a preset soft cue strategy to obtain attribute soft cue text features, object soft cue text features and combination soft cue text features.

[0011] The global semantic features, the attribute fine-grained semantic features, the object fine-grained semantic features, the combined fine-grained semantic features, the attribute soft-hint text features, the object soft-hint text features, and the combined soft-hint text features are input into the loss calculation module of the cross-modal semantic alignment model for image-text alignment. The alignment results are then linearly processed to obtain the alignment prediction results. During model training, the loss calculation module iteratively updates the model parameters through the backpropagation mechanism of the total loss function until the total loss function converges, thereby obtaining the trained cross-modal semantic alignment model.

[0012] In one embodiment, high-level visual feature extraction is performed on the image to be aligned based on a preset query vector to obtain the visual fine-grained semantic features corresponding to the image to be aligned, including:

[0013] The image to be aligned is sequentially encoded in the image encoder to obtain the corresponding global visual features, and high-level visual features are extracted in the high-level coding block of the image encoder.

[0014] The high-level visual features are used in the semantic feature extraction module to update the preset query vector pair in order to obtain the visual fine-grained semantic features.

[0015] In one embodiment, the high-level visual features are used in the semantic feature extraction module to update the preset query vector to obtain the fine-grained visual semantic features, including:

[0016] The visual features are fine-tuned based on the preset Adapter module: To obtain the finely tuned visual features ;in, This represents the high-level visual features output by the i-th high-level coding block;

[0017] The fine-tuned visual features are used as keys and values, the preset query vector is used as the query, and the preset query vector is updated through a preset cross-attention mechanism.

[0018] The visual fine-grained semantic features are determined based on the updated query vector.

[0019] In one embodiment, the semantic feature decoupling module consists of three sets of pre-defined Transformer encoders with consistent structure and independent parameters, and a pre-defined projection layer;

[0020] The visual fine-grained semantic features are input into the semantic feature decoupling module of the cross-modal semantic alignment model for decoupling processing to obtain attribute fine-grained semantic features, object fine-grained semantic features, and combined fine-grained semantic features, including:

[0021] Based on the preset Transformer encoder and the preset projection layer, the visual fine-grained semantic features are sequentially encoded, normalized, and projected in the semantic feature decoupling module to obtain the attribute fine-grained semantic features, the object fine-grained semantic features, and the combined fine-grained semantic features.

[0022] In one embodiment, the visual fine-grained semantic features are sequentially encoded, normalized, and projected using the following formula:

[0023] ;

[0024] ;

[0025] ;

[0026] in, Represents fine-grained semantic features of attributes; Represents fine-grained semantic features of objects; Representing the combination of fine-grained semantic features: Indicates the preset attribute projection layer; Indicates the preset object projection layer; Indicates a preset combined projection layer; Each represents a single preset Transformer encoder; B represents the normalization layer; B represents the visual fine-grained semantic features. Let represent the D-dimensional real space.

[0027] In one embodiment, the image to be aligned and text labels corresponding to attributes, objects, and combinations in the image to be aligned are input into the text encoder of the cross-modal semantic alignment model, and text feature extraction is performed based on a preset soft cueing strategy, including:

[0028] The attribute text labels and object text labels are word-embedding processed by a pre-trained CLIP word embedding encoder to obtain attribute word embedding features corresponding to the attribute text labels and object word embedding features corresponding to the object text labels.

[0029] Based on the attribute word embedding features and the object word embedding features, attribute soft hint features, object soft hint features, and combined soft hint features are determined;

[0030] The attribute soft-hint feature, the object soft-hint feature, and the combined soft-hint feature are input into the text encoding to extract text features, thereby obtaining the corresponding attribute soft-hint text features, the object soft-hint text features, and the combined soft-hint text features.

[0031] In one embodiment, the global semantic features, the attribute fine-grained semantic features, the object fine-grained semantic features, the combined fine-grained semantic features, the attribute soft-hint text features, the object soft-hint text features, and the combined soft-hint text features are input into the loss calculation module of the cross-modal semantic alignment model for image-text alignment, and the alignment results are linearly processed to obtain the alignment prediction results, including:

[0032] The attribute compatibility prediction score is determined based on the fine-grained semantic features of the attribute and the soft-hint text features of the attribute.

[0033] The object compatibility prediction score is determined based on the fine-grained semantic features of the object and the soft-hint text features of the object.

[0034] The global compatibility prediction score is determined based on the global semantic features and the combined soft prompt text features.

[0035] The combined compatibility prediction score is determined based on the combined fine-grained semantic features and the combined soft prompt text features.

[0036] The attribute compatibility prediction score, the object compatibility prediction score, the global compatibility prediction score, and the combined compatibility prediction score are linearly processed to obtain the alignment prediction result.

[0037] In one embodiment, the cross-modal semantic alignment method based on fine-grained semantic decoupling further includes:

[0038] The total loss function is determined according to the following steps:

[0039] Based on the training attribute compatibility prediction scores of the training images in the training set, determine the attribute loss function;

[0040] The object loss function is determined based on the training object compatibility prediction scores of the training images in the training set.

[0041] The global loss function is determined based on the global compatibility prediction scores of the training images in the training set.

[0042] Determine the combination loss function based on the training combination compatibility prediction scores of the training images in the training set.

[0043] The total loss function is determined based on the attribute loss function, the object loss function, the global loss function, and the combined loss function.

[0044] Embodiments of the present invention also provide a cross-modal semantic alignment device based on fine-grained semantic decoupling, comprising:

[0045] The acquisition module is used to acquire the image to be aligned;

[0046] The processing module is used to input the image to be aligned into a pre-trained image encoder of a cross-modal semantic alignment model to obtain global semantic features; to use a semantic feature extraction module in the high-level coding block of the image encoder to perform high-level visual feature extraction on the image to be aligned based on a preset query vector to obtain visual fine-grained semantic features corresponding to the image to be aligned; to input the visual fine-grained semantic features into the semantic feature decoupling module of the cross-modal semantic alignment model for decoupling processing to obtain attribute fine-grained semantic features, object fine-grained semantic features, and combined fine-grained semantic features; and to input the image to be aligned and the text labels corresponding to the attributes, objects, and combinations in the image to be aligned into the text encoder of the cross-modal semantic alignment model. In the process, text features are extracted based on a preset soft-hint strategy to obtain attribute soft-hint text features, object soft-hint text features, and combined soft-hint text features. The global semantic features, attribute fine-grained semantic features, object fine-grained semantic features, combined fine-grained semantic features, attribute soft-hint text features, object soft-hint text features, and combined soft-hint text features are input into the loss calculation module of the cross-modal semantic alignment model for image-text alignment, and the alignment results are linearly processed to obtain the alignment prediction result. During the model training process, the loss calculation module iteratively updates the model parameters through the backpropagation mechanism of the total loss function until the total loss function converges to obtain the trained cross-modal semantic alignment model.

[0047] Embodiments of the present invention also provide a computing device, comprising:

[0048] Memory, used to store one or more programs;

[0049] One or more processors are configured to execute the one or more programs to implement the method described above.

[0050] The above-described solution of the present invention has at least the following beneficial effects:

[0051] The cross-modal semantic alignment method, apparatus, and device based on fine-grained semantic decoupling provided by the above-described solution of the present invention obtains global semantic features by inputting the acquired image to be aligned into a pre-trained image encoder of a cross-modal semantic alignment model. A semantic feature extraction module is used in the high-level coding block of the image encoder to extract high-level visual features of the image to be aligned based on a preset query vector, thereby obtaining the visual fine-grained semantic features corresponding to the image to be aligned. The visual fine-grained semantic features are then input into the semantic feature decoupling module of the cross-modal semantic alignment model for decoupling processing to obtain attribute fine-grained semantic features, object fine-grained semantic features, and combined fine-grained semantic features. Finally, the image to be aligned and the text labels corresponding to the attributes, objects, and combinations in the image to be aligned are input into... In the text encoder of the cross-modal semantic alignment model, text features are extracted based on a preset soft-cue strategy to obtain attribute soft-cue text features, object soft-cue text features, and combined soft-cue text features. These global semantic features, attribute fine-grained semantic features, object fine-grained semantic features, combined fine-grained semantic features, attribute soft-cue text features, object soft-cue text features, and combined soft-cue text features are input into the loss calculation module of the cross-modal semantic alignment model for image-text alignment. The alignment results are then linearly processed to obtain the alignment prediction result. During model training, the loss calculation module iteratively updates the model parameters through the backpropagation mechanism of the total loss function until the total loss function converges, thus obtaining the trained cross-modal semantic alignment model. Based on fine-grained semantic decoupling, high-level visual features are utilized to address the problem of insufficient decoupling of attribute and object semantic concepts caused by the limited global visual feature representation capability of CLIP image encoding. Alignment based on the decoupled visual features and cue text features significantly improves the model's generalization ability to unseen combinations, thereby improving the accuracy and efficiency of cross-modal semantic alignment. Attached Figure Description

[0052] Figure 1 This is a flowchart of the cross-modal semantic alignment method based on fine-grained semantic decoupling provided in this embodiment of the invention;

[0053] Figure 2This is an overall framework diagram of a cross-modal semantic alignment model provided in an optional embodiment of the present invention;

[0054] Figure 3 This is a block diagram of a cross-modal semantic alignment device based on fine-grained semantic decoupling provided in an embodiment of the present invention;

[0055] Figure 4 This is a schematic block diagram of an electronic device provided in an embodiment of the present invention;

[0056] Figure 5 This is a schematic block diagram of a computing device provided in an embodiment of the present invention. Detailed Implementation

[0057] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0058] In the following description, certain specific details are set forth for the purpose of illustrating various disclosed embodiments in order to provide a thorough understanding of the various disclosed embodiments. However, those skilled in the art will recognize that embodiments may be practiced without one or more of these specific details. In other instances, well-known apparatuses, structures, and techniques associated with this application may not have been shown or described in detail to avoid unnecessarily obscuring the description of the embodiments.

[0059] Throughout this specification, references to "an embodiment" or "an embodiment" indicate that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Therefore, the appearance of "in an embodiment" or "an embodiment" in various places throughout the specification does not necessarily refer to the same embodiment. Furthermore, a particular feature, structure, or characteristic may be combined in any manner in one or more embodiments.

[0060] like Figure 1 As shown, embodiments of the present invention provide a cross-modal semantic alignment method based on fine-grained semantic decoupling, comprising:

[0061] Step 11, obtain the image to be aligned;

[0062] Step 12: Input the image to be aligned into the pre-trained image encoder of the cross-modal semantic alignment model to obtain global semantic features. Use the semantic feature extraction module in the high-level coding block of the image encoder and perform high-level visual feature extraction on the image to be aligned based on the preset query vector to obtain the visual fine-grained semantic features corresponding to the image to be aligned.

[0063] Step 13: Input the visual fine-grained semantic features into the semantic feature decoupling module of the cross-modal semantic alignment model for decoupling processing to obtain attribute fine-grained semantic features, object fine-grained semantic features and combined fine-grained semantic features.

[0064] Step 14: Input the image to be aligned and the text labels corresponding to the attributes, objects and combinations in the image to be aligned into the text encoder of the cross-modal semantic alignment model, and extract text features based on the preset soft cue strategy to obtain attribute soft cue text features, object soft cue text features and combination soft cue text features.

[0065] Step 15: Input the global semantic features, attribute fine-grained semantic features, object fine-grained semantic features, combined fine-grained semantic features, attribute soft-hint text features, object soft-hint text features, and combined soft-hint text features into the loss calculation module of the cross-modal semantic alignment model for image-text alignment, and perform linear processing on the alignment results to obtain the alignment prediction results; During the model training process, the loss calculation module iteratively updates the model parameters through the backpropagation mechanism of the total loss function until the total loss function converges to obtain the trained cross-modal semantic alignment model.

[0066] In this embodiment, the cross-modal semantic alignment model FGSD-CZSL mainly consists of four parts: an image encoder. (Including the parallel semantic feature extraction module SFEM), the semantic feature disentanglement module SFDM, and the text encoder) And a loss calculation module;

[0067] like Figure 2 As shown, a pre-trained CLIP image encoder based on the ViT (Vision Transformer) architecture can be used as the image encoder in the cross-modal semantic alignment model FGSD-CZSL. The backbone network of the image encoder Composed of multiple Transformer coding blocks, it possesses powerful visual representation capabilities; among them, the semantic feature extraction module SFEM is embedded in the image encoder. Within the high-level transformer coding block and running synchronously with it, the semantic feature extraction module SFEM captures fine-grained visual semantic features of the image to be aligned based on preset query vectors, thereby enhancing the model's ability to capture fine-grained visual semantic information. Here, the preset query vectors are a set of learnable vectors that are updated during the iterative training of the model. Preferably, the number of vectors in the preset query vectors should be at a moderate level to avoid performance degradation due to model overfitting caused by too many vectors, or insufficient semantic expression during high-level visual feature extraction due to too few vectors, resulting in mediocre model performance.

[0068] The Semantic Feature Decoupling Module (SFDM) is used to decouple attributes and objects in fine-grained visual semantic features; in the image encoder In this process, by extracting and decoupling high-level visual features, the problem of insufficient decoupling between attributes and object semantic concepts caused by the limited global visual feature representation capability of CLIP image encoding is solved, so as to achieve full decoupling and ensure the accuracy of subsequent modality alignment.

[0069] Text encoder It is a pre-trained CLIP text encoder that extracts text features based on the input attributes, objects, and corresponding prompt words, and a preset soft prompting strategy.

[0070] Furthermore, based on the loss calculation module, the fine-grained semantic features of the image to be aligned are aligned with each branch of the soft prompt text features. By meticulously extracting the visual semantic features of the image and modeling their branches, and aligning them with the extracted text prompt features, the model's generalization ability to unseen combinations is significantly improved, thereby ensuring the accuracy of cross-modal semantic alignment.

[0071] It should be noted that the training process of the model is the same as the actual processing process. The difference is that during training, the model parameters are iteratively updated through the backpropagation mechanism of the constructed total loss function. Here, the total loss function includes four branch loss functions: attribute loss function, object loss function, global loss function, and combined loss function. The training continues until the total loss function converges to obtain the trained cross-modal semantic alignment model.

[0072] In an optional embodiment of the present invention, step 12 above may include:

[0073] Step 121: The images to be aligned are sequentially encoded in the lower-level coding blocks of the image encoder to obtain the corresponding global visual features, and the global visual features are extracted in the higher-level coding blocks of the image encoder to obtain the higher-level visual features.

[0074] Step 122: In the semantic feature extraction module, the preset query vector is updated based on the high-level visual features to obtain fine-grained visual semantic features.

[0075] In this embodiment, for the input image to be aligned Where H represents the image height; W represents the image width; and C represents the number of image channels. First, in the image encoder... Feature extraction is performed to obtain global semantic features. Secondly, in the image encoder Feature extraction is performed on the lower-level transformer coding blocks (the transformer coding blocks that appear earlier in the sequence of multiple transformer coding blocks) to obtain the hidden state features. This hidden state feature Further feature extraction is performed in the remaining high-level transformer coding blocks, and the visual features are output after extraction in the i-th layer transformer coding block. ;

[0076] Furthermore, step 122 above may include:

[0077] Step 1221: Fine-tune the visual features based on the preset Adapter module: To obtain the finely tuned visual features And input it into the next layer of coding block in the form of residuals; where, This represents the visual features output by the i-th higher-level coded block;

[0078] Step 1222: Using the fine-tuned visual features as keys and values, and the preset query vector as the query, the preset query vector is updated using a preset cross-attention mechanism; the specific update process is as follows:

[0079] ;

[0080] in, Indicates the preset query vector; This represents the updated query vector; , , These represent the weight matrices for Query, Key, and Value, respectively; T represents the matrix transpose. Indicates the scaling factor;

[0081] Step 1223: Determine the fine-grained visual semantic features based on the updated query vector; specifically, this can be determined using the following formula:

[0082] ;

[0083] in, Represents fine-grained semantic features of the visual representation; Indicates the normalization layer; This represents the feedforward layer; compared to a single global visual feature. Visual fine-grained semantic features It can perform visual semantic representation from multiple dimensions, solving the problem of insufficient decoupling between attribute and object semantic concepts caused by the limited global visual feature representation capability of CLIP image encoding.

[0084] In an optional embodiment of the present invention, the semantic feature decoupling module consists of three sets of Transformer encoders with consistent structure and independent parameters, and a preset projection layer;

[0085] Step 13 above may include:

[0086] Step 131: Based on the preset Transformer encoder and preset projection layer, the visual fine-grained semantic features are encoded, normalized and projected sequentially in the semantic feature decoupling module to obtain attribute fine-grained semantic features, object fine-grained semantic features and combined fine-grained semantic features.

[0087] In this embodiment, a preset Transformer encoder, a normalization layer, and a preset projection layer are used to decouple the fine-grained semantic features of the vision, so as to improve the model's understanding of detailed semantic concepts and ensure the accuracy of subsequent decoupling. Here, the preset Transformer encoder can efficiently extract features through multi-head self-attention, residual connections, and feedforward networks.

[0088] In an optional embodiment of the present invention, the visual fine-grained semantic features are sequentially encoded, normalized, and projected using the following formula:

[0089] ;

[0090] ;

[0091] ;

[0092] in, Represents fine-grained semantic features of attributes; Represents fine-grained semantic features of objects; Represents the combination of fine-grained semantic features: Indicates the preset attribute projection layer; Indicates the preset object projection layer; Indicates a preset combined projection layer; Each represents a single preset Transformer encoder; Indicates the normalization layer; Represents fine-grained semantic features of the visual representation; Let represent the D-dimensional real space.

[0093] In an optional embodiment of the present invention, step 14 above may include:

[0094] Step 141: The attribute text labels and object text labels are embedded and encoded by a pre-trained CLIP word embedding encoder to obtain the attribute word embedding features corresponding to the attribute text labels and the object word embedding features corresponding to the object text labels.

[0095] Step 142: Based on attribute word embedding features and object word embedding features, determine attribute soft cue features, object soft cue features, and combined soft cue features;

[0096] Step 143: Input the attribute soft cue features, object soft cue features, and combined soft cue features into the text encoding to extract text features, so as to obtain the corresponding attribute soft cue text features, object soft cue text features, and combined soft cue text features.

[0097] In this embodiment, both attribute text labels and object text labels are collections of text labels, which can be represented as follows: , ;in, These all represent attribute text tags, such as color attribute text tags like "white" and "black". These all represent object text tags, such as animal object text tags like "cat" and "dog". and These represent the total number of attributes and objects in the dataset corresponding to the image to be aligned, respectively.

[0098] Here, you can add attribute text labels. With object text label collection Word embeddings are performed using a pre-trained CLIP word embedding encoder, resulting in the corresponding attribute word embedding feature set. (Contains multiple attribute word embedding features) and object word embedding feature set (Includes multiple object word embedding features):

[0099] ;

[0100] ;

[0101] in: Both represent attribute word embedding features; Both represent object word embedding features;

[0102] Furthermore, the following soft-suggestion features are constructed for all attribute word embedding features, object word embedding features, and combined embedding features using a preset soft-suggestion strategy:

[0103] ;

[0104] ;

[0105] ;

[0106] in, Indicates the characteristics of soft-suggestion text; The soft-suggestion text features of the object; This indicates the combined features of soft prompt text; All represent learnable cue word prefix embedding features initialized using the image to be aligned; i=1,2,...,n; j=1,2,...,m.

[0107] Furthermore, the attribute soft hint feature Soft cue features of objects Combined soft cue features Input into the text encoder respectively Text feature extraction is performed to obtain the corresponding soft cue text features:

[0108] ;

[0109] ;

[0110] ;

[0111] in, Indicates the characteristics of soft-suggestion text; The soft-suggestion text features of the object; This indicates the combination of soft prompt text features.

[0112] In an optional embodiment of the present invention, step 15 above may include:

[0113] Step 151: Determine the attribute compatibility prediction score based on the fine-grained semantic features and the soft-hint text features of the attributes; specifically, it can be determined using the following formula:

[0114] ;

[0115] in, Indicates the attribute compatibility prediction score; This represents the learnable temperature parameter; k1 represents the attribute label index; Represents a collection of attribute text labels.

[0116] Step 152: Determine the object compatibility prediction score based on the object's fine-grained semantic features and the object's soft-hint text features; specifically, it can be determined using the following formula:

[0117]

[0118] in, Indicates the object compatibility prediction score; k2 represents the learnable temperature parameter; k2 represents the object label index. Represents a collection of text labels for an object.

[0119] Step 153: Determine the global compatibility prediction score based on the global semantic features and the combined soft-hint text features; specifically, it can be determined using the following formula:

[0120] ;

[0121] in, Indicates the global compatibility prediction score; This represents the learnable temperature parameter; k3 represents the combined label index; Represents global semantic features (extracted via an image encoder): , (Indicates the image to be aligned); This represents a set of combined text tags.

[0122] Step 154: Determine the combined compatibility prediction score based on the combined fine-grained semantic features and the combined soft-hint text features; specifically, it can be determined using the following formula:

[0123] ;

[0124] in, k3 represents the combined compatibility prediction score; k3 represents the combined label index. This represents a learnable temperature parameter; This represents a set of combined text tags.

[0125] Step 155 involves linearly processing the attribute compatibility prediction score, object compatibility prediction score, global compatibility prediction score, and combined compatibility prediction score to obtain the alignment prediction result; specifically, it can be expressed by the following formula:

[0126]

[0127] in, This indicates the prediction result (i.e., the predicted classification score). This represents the weighting coefficient used to control the proportion of the global compatibility prediction score to the combined compatibility prediction score. These are tags for test set combinations.

[0128] In an optional embodiment of the present invention, the above-described cross-modal semantic alignment method based on fine-grained semantic decoupling may further include:

[0129] Step 16: Determine the total loss function according to the following steps:

[0130] Step 161: Determine the attribute loss function based on the training attribute compatibility prediction scores of the training images in the training set;

[0131] Step 162: Determine the object loss function based on the training object compatibility prediction scores of the training images in the training set;

[0132] Step 163: Determine the global loss function based on the global compatibility prediction scores of the training images in the training set;

[0133] Step 164: Determine the combination loss function based on the training combination compatibility prediction scores of the training images in the training set;

[0134] Step 165: Determine the total loss function based on the attribute loss function, object loss function, global loss function, and combined loss function.

[0135] In this embodiment, the training of attribute compatibility prediction score, object compatibility prediction score, global compatibility prediction score, and combined compatibility prediction score are solved in the same way as the above-mentioned attribute compatibility prediction score, object compatibility prediction score, global compatibility prediction score, and combined compatibility prediction score, and will not be repeated here.

[0136] The attribute loss function is obtained based on the training attribute compatibility prediction score and the training object compatibility prediction score, respectively. and object functions :

[0137] ;

[0138] ;

[0139] in, This represents the training attribute compatibility prediction score; Indicates the training object compatibility prediction score; This represents the training set.

[0140] The global loss function is obtained based on the training global compatibility prediction score and the training combined compatibility prediction score, respectively. and combined loss function :

[0141] ;

[0142] ;

[0143] in, This represents the global compatibility prediction score during training. Indicates the training portfolio compatibility prediction score; This represents the training set.

[0144] Furthermore, the attribute loss function, object loss function, global loss function, and combined loss function are combined to form the overall loss function. :

[0145] ;

[0146] in, This represents the weighting coefficients used to stabilize the model's training process.

[0147] The following evaluation of the methods provided in the above embodiments will be conducted on three CZSL benchmark datasets: the C-GQA dataset, the UT-Zappos dataset, and the MIT-States dataset. The C-GQA dataset contains 39,298 images, covering a vast number of objects and attribute categories of common real-world concepts, with a total of 7,767 attribute-object combinations annotated, making it the largest benchmark dataset in the field of zero-step combinatorial learning. The UT-Zappos dataset contains 50,025 images of footwear products, focusing on fine-grained recognition, covering 12 footwear categories and 16 material attributes. Its main challenge lies in the extremely similar visual features of materials such as "artificial leather" and "genuine leather," requiring the model to have the ability to capture subtle textures and material differences. The MIT-States dataset contains 53,753 images of everyday scenes, covering 115 attributes and 245 objects, forming 1,962 attribute-object combinations.

[0148] The three datasets are divided as shown in Table 1 below. The training set contains all known combinations of attributes and objects, while the validation and test sets contain some known and unseen combinations. Indicates the number of attributes. Indicates the number of objects. Indicates the number of seen combinations; This indicates that no combination quantity was found. Indicates the number of images.

[0149] Table 1. Partitioning of the CZSL benchmark dataset

[0150]

[0151] Evaluation metrics include: best accuracy with known combinations (S), best accuracy without known combinations (U), best harmonic mean (HM), and area under the curve (AUC). Among these, AUC is the highest priority metric because it provides a comprehensive evaluation of the model.

[0152] The comparison methods include:

[0153] (1) CSP used CLIP in the CZSL task to replace the categories in the text prompts with trainable attributes and object tags.

[0154] (2) DFSP constructs attribute-object joint representation through soft prompts and achieves cross-modal decomposition and fusion in the language feature space, which effectively improves the ability to recognize unseen combinations.

[0155] (3) GIPCOL injects attribute-object combination graphs into soft cues, thereby enhancing the combination zero-shot generalization ability of the visual-language model through structured cues.

[0156] (4) CDS-CZSL quantifies the specificity of attributes by evaluating the applicability diversity and contextual relevance of attributes on different objects, and focuses on screening more informative attribute-object combinations.

[0157] (5) Troika proposed designing a cross-modal traction module to align branch-specific cue representations with decomposed visual features, effectively calibrating multimodal representation bias.

[0158] (6) PLID uses a large language model to construct a diverse and information-rich language-driven category distribution, and dynamically fuses the classification decision of the combination space and the primitive space through the visual-language primitive decomposition module.

[0159] (7) RAPR proposes a retrieval enhancement method to identify unseen combinations by retrieving and utilizing relevant original (attribute and object) representations from a training image database.

[0160] (8) MSCI uses a multi-stage interaction mechanism to extract low-level local features and high-level global features of the CLIP visual encoder by using two adaptive aggregators, and dynamically adjusts their attention weights in text representation, thereby enhancing the perception of fine-grained attribute-object combination.

[0161] (9) LOGICZSL formalizes the attribute-object semantic relationship mined by large language models into logical rules, and injects these rules into the existing CZSL method through logical guided loss to explicitly model rich combinatorial semantics.

[0162] The cross-modal semantic alignment model of this invention uses a pre-trained CLIP ViT-L / 14 model as the backbone network for image and text encoders, and is implemented using PyTorch. The model is trained on a single NVIDIA RTX4090 GPU for 10 epochs each on three public benchmark datasets (MIT-States, UT-Zappos, and C-GQA), with Adam as the optimizer. The weight coefficients are used to determine the optimal model. To balance the importance of global semantics and compositional semantics: In a closed-world setting, the semantics of MIT-States, UT-Zappos, and C-GQA are compared. The values ​​are set to 0.85, 0.50, and 0.85; in the open-world setting, they are 0.85, 0.50, and 0.85. For open-world evaluation, a feasibility score is introduced as a screening threshold to eliminate unreasonable attribute-object combinations, reduce the search space, and improve the model's generalization ability.

[0163] Table 2 shows the comparison results of different methods tested in a closed-world scenario. As can be seen from Table 2, the method of this invention achieved the best results across all metrics on the three datasets. Regarding AUC, the primary metric for comprehensive model evaluation, the method of this invention outperforms the Logic-Troika method by 1.8%, 3.5%, and 11.8% on MIT-States, UT-Zappos, and C-GQA datasets, respectively. Particularly for C-GQA, a dataset with a large number of attributes, objects, and complex combinations, the method outperforms the Logic-Troika method by 4.1%, 5.1%, and 7.1% on S, U, and HM datasets, respectively. This comprehensive improvement is attributed to FGSD-CZSL's ability to effectively decouple attribute and object concepts with fine granularity.

[0164] Table 2. Comparison results of different methods tested in a closed-world scenario.

[0165]

[0166] Table 3 shows the comparison results of different methods tested in open-world scenarios. As can be seen from Table 3, the performance of each method decreases after transitioning from a closed world to a more realistic open world, especially for recognizing unseen combinations. However, the method of this invention still achieves the best performance in open-world scenarios on three benchmark datasets (MIT-States, UT-Zappos, and C-GQA). The method of this invention achieves the best performance with AUCs of 8.9%, 38.6%, and 4.6% on the three datasets, respectively. In particular, on the fine-grained dataset UT-Zappos, it outperforms the Logic-Troika method by 2.6% and 1.4% in both visible and invisible combination accuracy, demonstrating the effectiveness of the method in fine-grained combination recognition and its ability to effectively decouple attributes from objects. Furthermore, the method of this invention also performs excellently on C-GQA, with an AUC significantly better than other methods (e.g., Troika's AUC=2.7), indicating that the model can effectively distinguish and generalize complex semantic concepts.

[0167] Table 3. Comparison results of different methods tested in open-world scenarios.

[0168]

[0169] The cross-modal semantic alignment method based on fine-grained semantic decoupling provided by the above embodiments of the present invention obtains global semantic features by inputting the acquired image to be aligned into a pre-trained image encoder of the cross-modal semantic alignment model. A semantic feature extraction module is used in the high-level coding block of the image encoder to extract high-level visual features of the image to be aligned based on a preset query vector, thereby obtaining the visual fine-grained semantic features corresponding to the image to be aligned. The visual fine-grained semantic features are then input into the semantic feature decoupling module of the image encoder for decoupling processing to obtain attribute fine-grained semantic features, object fine-grained semantic features, and combined fine-grained semantic features. Finally, the image to be aligned and the text labels corresponding to the attributes, objects, and combinations in the image to be aligned are input into the cross-modal semantic alignment model. In the text encoder of the semantic alignment model, text features are extracted based on a preset soft-cue strategy to obtain attribute soft-cue text features, object soft-cue text features, and combined soft-cue text features. Global semantic features, attribute fine-grained semantic features, object fine-grained semantic features, combined fine-grained semantic features, attribute soft-cue text features, object soft-cue text features, and combined soft-cue text features are input into the loss calculation module of the cross-modal semantic alignment model for image-text alignment. The scores obtained from the alignment results are linearly processed to obtain the alignment prediction result. During model training, the loss calculation module iteratively updates the model parameters through the backpropagation mechanism of the total loss function until the total loss function converges, thus obtaining the trained cross-modal semantic alignment model. Based on fine-grained semantic decoupling, that is, utilizing high-level visual features, it solves the problem of insufficient decoupling of attribute and object semantic concepts caused by the limited global visual feature representation capability of CLIP image encoding. Simultaneously, through detailed extraction of image semantic features and its branch modeling and cue representation alignment, the model's generalization ability to unseen combinations is significantly improved, thereby improving the accuracy and efficiency of cross-modal semantic alignment.

[0170] like Figure 3 As shown, embodiments of the present invention also provide a cross-modal semantic alignment device 30 based on fine-grained semantic decoupling, comprising:

[0171] Module 31 is used to acquire the image to be aligned;

[0172] Processing module 32 is used to input the image to be aligned into the pre-trained image encoder of the cross-modal semantic alignment model to obtain global semantic features. In the high-level coding block of the image encoder, a semantic feature extraction module is used to extract high-level visual features of the image to be aligned based on a preset query vector, thereby obtaining the visual fine-grained semantic features corresponding to the image to be aligned. The visual fine-grained semantic features are then input into the semantic feature decoupling module of the image encoder for decoupling processing to obtain attribute fine-grained semantic features, object fine-grained semantic features, and combined fine-grained semantic features. Finally, the image to be aligned and the text labels corresponding to the attributes, objects, and combinations in the image to be aligned are input into the text encoder of the cross-modal semantic alignment model. In the process, text features are extracted based on a preset soft-cue strategy to obtain attribute soft-cue text features, object soft-cue text features, and combined soft-cue text features. The global semantic features, attribute fine-grained semantic features, object fine-grained semantic features, combined fine-grained semantic features, attribute soft-cue text features, object soft-cue text features, and combined soft-cue text features are input into the loss calculation module of the cross-modal semantic alignment model for image-text alignment. The alignment results are linearly processed to obtain the alignment prediction results. During the model training process, the loss calculation module iteratively updates the model parameters through the backpropagation mechanism of the total loss function until the total loss function converges to obtain the trained cross-modal semantic alignment model.

[0173] It should be noted that this device is the same as the cross-modal semantic alignment method based on fine-grained semantic decoupling described above. All implementation methods in the above method embodiments are applicable to this embodiment and can achieve the same technical effect.

[0174] like Figure 4 As shown, embodiments of the present invention also provide an electronic device 50, comprising: a memory 51 for storing one or more computer programs; and one or more processors 52 for executing the one or more computer programs. When the computer programs are run by the processors, they perform the cross-modal semantic alignment method based on fine-grained semantic decoupling as described above. All implementations in the above method embodiments are applicable to this embodiment and can achieve the same technical effects. The electronic device 50 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components, their connections and relationships, and their functions shown in this invention are merely examples and are not intended to limit the implementation of the invention described and / or claimed herein.

[0175] like Figure 5As shown, electronic device 50 is a computing device or computer system, which may include CPU 501 (computing unit), which can perform various appropriate actions and processes according to a computer program stored in ROM 502 (read-only memory) or a computer program loaded from storage unit 508 into random access RAM 503 (memory). RAM 503 may also store various programs and data required for the operation of device 500. CPU 501, ROM 502, and RAM 503 are interconnected via bus 504. I / O interface 505 (input / output interface) is also connected to bus 504.

[0176] Multiple components in electronic device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0177] CPU 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of CPU 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. CPU 501 performs the various methods and processes described above. For example, in some embodiments, the cross-modal semantic alignment method based on fine-grained semantic decoupling can be implemented as a computer software program tangibly contained in a computer-readable storage medium, such as storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by CPU 501, one or more steps of the cross-modal semantic alignment method based on fine-grained semantic decoupling described above can be performed. Alternatively, in other embodiments, CPU 501 may be configured by any other suitable means (e.g., by means of firmware) to perform a cross-modal semantic alignment method based on fine-grained semantic decoupling.

[0178] Embodiments of the present invention also provide a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the cross-modal semantic alignment method based on fine-grained semantic decoupling as described above. All implementations in the above method embodiments are applicable to this embodiment and can achieve the same technical effects.

[0179] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0180] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0181] In the embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0182] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0183] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0184] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0185] Furthermore, it should be noted that in the apparatus and method of the present invention, it is obvious that the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered equivalent solutions of the present invention. Moreover, the steps performing the above series of processes can naturally be executed in the order described, but are not necessarily required to be executed in chronological order; some steps can be executed in parallel or independently of each other. Those skilled in the art will understand that all or any step or component of the method and apparatus of the present invention can be implemented in any computing device (including processors, storage media, etc.) or network of computing devices, in hardware, firmware, software, or a combination thereof. This is something that those skilled in the art can achieve by using their basic programming skills after reading the description of the present invention.

[0186] Therefore, the object of the present invention can also be achieved by running a program or a set of programs on any computing device. The computing device can be a known general-purpose device. Therefore, the object of the present invention can also be achieved simply by providing a program product containing program code for implementing the method or apparatus. That is, such a program product also constitutes the present invention, and the storage medium storing such a program product also constitutes the present invention. Obviously, the storage medium can be any known storage medium or any storage medium developed in the future. It should also be noted that in the apparatus and method of the present invention, it is obvious that the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered equivalent to the present invention. Furthermore, the steps for performing the above series of processes can naturally be performed in the order described, but are not necessarily required to be performed in chronological order. Some steps can be performed in parallel or independently of each other.

[0187] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A cross-modal semantic alignment method based on fine-grained semantic decoupling, characterized in that, include: Obtain the image to be aligned; The image to be aligned is input into the pre-trained image encoder of the cross-modal semantic alignment model to obtain global semantic features. At the same time, a semantic feature extraction module is used in the high-level coding block of the image encoder, and high-level visual features are extracted from the image to be aligned based on a preset query vector to obtain the visual fine-grained semantic features corresponding to the image to be aligned. The visual fine-grained semantic features are input into the semantic feature decoupling module of the cross-modal semantic alignment model for decoupling processing to obtain attribute fine-grained semantic features, object fine-grained semantic features, and combined fine-grained semantic features. The image to be aligned and the text labels corresponding to the attributes, objects and combinations in the image to be aligned are input into the text encoder of the cross-modal semantic alignment model, and text features are extracted based on a preset soft cue strategy to obtain attribute soft cue text features, object soft cue text features and combination soft cue text features. The global semantic features, the attribute fine-grained semantic features, the object fine-grained semantic features, the combined fine-grained semantic features, the attribute soft-hint text features, the object soft-hint text features, and the combined soft-hint text features are input into the loss calculation module of the cross-modal semantic alignment model for image-text alignment, and the alignment results are linearly processed to obtain the alignment prediction results. During model training, the loss calculation module iteratively updates the model parameters through the backpropagation mechanism of the total loss function until the total loss function converges, thereby obtaining a trained cross-modal semantic alignment model. The semantic feature decoupling module consists of three sets of pre-defined Transformer encoders and pre-defined projection layers with consistent structures and independent parameters. The visual fine-grained semantic features are input into the semantic feature decoupling module of the cross-modal semantic alignment model for decoupling processing to obtain attribute fine-grained semantic features, object fine-grained semantic features, and combined fine-grained semantic features, including: Based on the preset Transformer encoder and the preset projection layer, the visual fine-grained semantic features are sequentially encoded, normalized, and projected in the semantic feature decoupling module to obtain the attribute fine-grained semantic features, the object fine-grained semantic features, and the combined fine-grained semantic features.

2. The cross-modal semantic alignment method based on fine-grained semantic decoupling according to claim 1, characterized in that, High-level visual features are extracted from the image to be aligned based on a preset query vector to obtain fine-grained visual semantic features corresponding to the image to be aligned, including: The image to be aligned is sequentially encoded in the image encoder to obtain the corresponding global visual features. At the same time, feature extraction is performed in the high-level coding block of the image encoder to obtain high-level visual features. In the semantic feature extraction module, the preset query vector pair is updated based on the high-level visual features to obtain the visual fine-grained semantic features.

3. The cross-modal semantic alignment method based on fine-grained semantic decoupling according to claim 1, characterized in that, The visual fine-grained semantic features are sequentially encoded, normalized, and projected using the following formula: ； ； ； in, Represents fine-grained semantic features of attributes; Represents fine-grained semantic features of objects; Represents the combination of fine-grained semantic features: Indicates the preset attribute projection layer; Indicates the preset object projection layer; Indicates a preset combined projection layer; Each represents a single preset Transformer encoder; B represents the normalization layer; B represents the visual fine-grained semantic features. Let represent the D-dimensional real space.

4. The cross-modal semantic alignment method based on fine-grained semantic decoupling according to claim 1, characterized in that, The image to be aligned and the text labels corresponding to the attributes, objects, and combinations in the image to be aligned are input into the text encoder of the cross-modal semantic alignment model, and text feature extraction is performed based on a preset soft cueing strategy, including: The attribute text labels and object text labels are word-embedding processed by a pre-trained CLIP word embedding encoder to obtain attribute word embedding features corresponding to the attribute text labels and object word embedding features corresponding to the object text labels. Based on the attribute word embedding features and the object word embedding features, attribute soft hint features, object soft hint features, and combined soft hint features are determined; The attribute soft-hint feature, the object soft-hint feature, and the combined soft-hint feature are input into the text encoding to extract text features, thereby obtaining the corresponding attribute soft-hint text features, the object soft-hint text features, and the combined soft-hint text features.

5. The cross-modal semantic alignment method based on fine-grained semantic decoupling according to claim 1, characterized in that, The global semantic features, the attribute fine-grained semantic features, the object fine-grained semantic features, the combined fine-grained semantic features, the attribute soft-hint text features, the object soft-hint text features, and the combined soft-hint text features are input into the loss calculation module of the cross-modal semantic alignment model for image-text alignment. The alignment results are then linearly processed to obtain the alignment prediction results, including: The attribute compatibility prediction score is determined based on the fine-grained semantic features of the attribute and the soft-hint text features of the attribute. The object compatibility prediction score is determined based on the fine-grained semantic features of the object and the soft-hint text features of the object. The global compatibility prediction score is determined based on the global semantic features and the combined soft suggestion text features. The combined compatibility prediction score is determined based on the combined fine-grained semantic features and the combined soft prompt text features. The attribute compatibility prediction score, the object compatibility prediction score, the global compatibility prediction score, and the combined compatibility prediction score are linearly processed to obtain the alignment prediction result.

6. The cross-modal semantic alignment method based on fine-grained semantic decoupling according to claim 5, characterized in that, Also includes: The total loss function is determined according to the following steps: Based on the training attribute compatibility prediction scores of the training images in the training set, determine the attribute loss function; The object loss function is determined based on the training object compatibility prediction scores of the training images in the training set. The global loss function is determined based on the global compatibility prediction scores of the training images in the training set. Determine the combination loss function based on the training combination compatibility prediction scores of the training images in the training set. The total loss function is determined based on the attribute loss function, the object loss function, the global loss function, and the combined loss function.

7. A cross-modal semantic alignment device based on fine-grained semantic decoupling, characterized in that, include: The acquisition module is used to acquire the image to be aligned; The processing module is used to input the image to be aligned into a pre-trained image encoder of a cross-modal semantic alignment model to obtain global semantic features. A semantic feature extraction module is used in the high-level coding block of the image encoder to extract high-level visual features of the image to be aligned based on a preset query vector, thereby obtaining fine-grained visual semantic features corresponding to the image to be aligned. The fine-grained visual semantic features are then input into the semantic feature decoupling module of the cross-modal semantic alignment model for decoupling processing to obtain fine-grained attribute semantic features, fine-grained object semantic features, and combined fine-grained semantic features. The image to be aligned and the text labels corresponding to the attributes, objects and combinations in the image to be aligned are input into the text encoder of the cross-modal semantic alignment model, and text features are extracted based on a preset soft cue strategy to obtain attribute soft cue text features, object soft cue text features and combination soft cue text features. The global semantic features, attribute fine-grained semantic features, object fine-grained semantic features, combined fine-grained semantic features, attribute soft-hint text features, object soft-hint text features, and combined soft-hint text features are input into the loss calculation module of the cross-modal semantic alignment model for image-text alignment, and the alignment results are linearly processed to obtain the alignment prediction results. During model training, the loss calculation module iteratively updates the model parameters through the backpropagation mechanism of the total loss function until the total loss function converges, thereby obtaining a trained cross-modal semantic alignment model. The semantic feature decoupling module consists of three sets of pre-defined Transformer encoders and pre-defined projection layers with consistent structures and independent parameters. The visual fine-grained semantic features are input into the semantic feature decoupling module of the cross-modal semantic alignment model for decoupling processing to obtain attribute fine-grained semantic features, object fine-grained semantic features, and combined fine-grained semantic features, including: Based on the preset Transformer encoder and the preset projection layer, the visual fine-grained semantic features are sequentially encoded, normalized, and projected in the semantic feature decoupling module to obtain the attribute fine-grained semantic features, the object fine-grained semantic features, and the combined fine-grained semantic features.

8. A computing device, characterized in that, include: Memory, used to store one or more programs; One or more processors are configured to execute the one or more programs to implement the method as described in any one of claims 1 to 6.

Images