A zero-shot learning method and system based on multi-scale semantic fine-grained embedding

Abstract

The application relates to the field of machine learning, and provides a zero-shot learning based on multi-scale semantic fine-grained embedding and a system, which comprises the following steps: step 1: adopting a frozen CLIP text encoder to process attribute, object and combined prompt texts; step 2: performing initialization extraction on the text fine-grained; and step 3: using multi-scale semantic fine-grained embedding and a prediction output unit to improve the semantic hierarchy modeling capability in a traditional text prototype three-branch architecture. The application can inject semantic fine-grained information trained through image traction in a text learning process, and can design differential enhancement strategies for different semantic hierarchies, so that high-precision identification of unobserved combinations is realized.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, specifically to a zero-shot learning method and system based on multi-scale semantic fine-grained embedding. Background Technology

[0002] Zero-shot learning, an important branch of machine learning, aims to enable models to recognize new categories not seen during the training phase. This is significant for solving practical problems such as long-tailed data distribution and rare category recognition. In recent years, with the rise of visual-language multimodal models (such as CLIP and ALIGN), compositional zero-shot learning has become a research hotspot. Its core idea is to decompose complex concepts into composable primitives (such as combinations of attributes and objects), and generalize to new combinatorial concepts by learning the representations of these primitives. For example, during the training phase, the model learns the concepts of "red apple" and "green truck," requiring it to recognize combinations of the unseen categories "red truck" or "green apple" during the inference phase. Its mainstream implementation schemes can be summarized as follows: (1) The image input image encoder obtains visual features; (2) Attribute-object composite text is input into a text encoder and mapped to text embedding features through text prompts (such as “a photo of a red truck”); (3) Both are trained in the shared embedding space using contrastive loss; (4) During the inference stage, the model completes zero-sample classification by calculating the cosine similarity between the image features and the combined text features of all candidates.

[0003] Thanks to the development of visual-language multimodal models such as CLIP, recent research has made significant progress in combinatorial zero-shot learning by exploring various enhancement methods using CLIP models. This work has effectively improved the alignment between combined labels and image features, enabling CLIP to perform well in combinatorial zero-shot learning tasks. However, the gap between different modalities still presents some urgent challenges and inherent problems for combinatorial zero-shot learning tasks.

[0004] (1) First, the visual discriminativeness of text-based representations is severely insufficient, leading to poor cross-modal alignment quality. Existing methods typically use category labels or attribute names optimized from text prompts as text input. This uniform scale of text representation fails to capture the richness and diversity of visual information. Image data contains a wealth of detailed information such as texture, shape, color distribution, spatial relationships, lighting conditions, and material properties, while simple text labels only provide abstract semantic concepts. This significant difference in the fine granularity of representation results in a substantial mismatch in information density between text features and visual features, severely limiting the accuracy of cross-modal alignment. Especially in scenarios where subtle differences need to be distinguished (such as "ripe fruit" versus "unripe fruit"), text representations lack sufficient discriminative cues, making it difficult for the model to establish accurate semantic-visual mapping relationships.

[0005] (2) Secondly, existing methods lack differentiated modeling mechanisms for different semantic levels. In combinatorial zero-shot learning tasks, attributes, objects, and their combinations represent three different semantic levels, and their needs for visual information are fundamentally different. Attributes (such as "red," "round," and "smooth") focus on local, fine-grained visual features, such as color distribution, texture patterns, and surface texture; objects (such as "car," "apple," and "chair") focus more on overall, structural, low-grained visual features, such as shape outlines, spatial layout, and component composition; while combinatorial concepts (such as "red car") require understanding the specific visual representation of attributes in the context of a particular object, involving the combinatorial interaction between attributes and objects. However, existing technologies usually adopt a uniform feature extraction and alignment strategy, failing to perform targeted optimization according to the characteristics of different semantic levels. This results in the learned text representations failing to achieve optimal discriminative ability at each level, thus affecting the alignment between text and images and failing to narrow the modal gap. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention aims to provide a zero-shot learning method and system based on multi-scale semantic fine-grained embedding. To solve these problems, this invention employs the following technical solutions: A zero-shot learning method based on multi-scale semantic fine-grained embedding includes the following steps: Step 1: Process the tooltip text of attributes, objects, and combinations using the frozen CLIP text encoder; Step 2: Perform fine-grained initial extraction of the text; Step 3: Enhance the semantic hierarchy modeling capability in the traditional text prototype three-branch architecture by leveraging multi-scale semantic fine-grained embedding and predictive output units.

[0007] Optionally, step 1 includes the following sub-steps: Step 1.1: The traditional text prototype three-branch module constructs three types of learnable hints: attributes, objects, and combinations. These hints are then encoded by the CLIP text encoder to obtain the corresponding level of text prototype features.

[0008] Optionally, step 1 includes the following sub-steps: Step 1.2: On the image side, the visual features are decomposed into independent features of attribute features and object features through the multi-head cross-attention decoupling module, so that the visual representation of attributes, objects and combinations is accurately aligned with the text prototype representation, thus solving the problem of entanglement between attributes and objects in visual features. During training, cross-modal attention scores between image and text prototypes are incorporated into similarity calculations to optimize the prediction probabilities of attributes and object branches.

[0009] Optionally, step 1 includes the following sub-steps: Step 1.3: Train the module using cross-entropy loss and update the parameters of the hint and decoupling modules.

[0010] Optionally, step two includes: In addition to the traditional three branches of the text prototype, three fine-grained branches for text prototype enhancement are introduced. The image-guided fine-grained text extraction module is responsible for extracting features from the input image to provide an information basis for subsequent semantic fine-grained learning and text prototype fine-grained enhancement. The initial attribute and object text fine-grained branches are the same as the attribute object branches of the text prototype. The frozen CLIP text encoder is used to process the cue text of attributes, objects and combinations, which serves as the basis for learning the subsequent image-guided text fine-grained extraction module. Unlike attribute and object text fine-grained branches, which are encoded by the encoder, the combined text fine-grained branches are obtained by using the initial attribute and text fine-grained branches through the initial attribute and object text fine-grained fusion mechanism to obtain the initial combined text fine-grained branches. The branches of fine-grained attribute and object text are initialized from the same source as the text prototype, while the combined fine-grained text is dynamically generated from the initial fine-grained text of attributes and objects through an innovative fusion mechanism; the process is as follows: First, for each attribute word in attribute set A (i represents the attribute word index, 1, 2, 3...) and each object word in the object set O. (j represents the word index, 1, 2, 3, ...), using a pre-trained CLIP text encoder. ( ) Extract its initial text fine-grained vector:

[0011] Where d is the feature dimension; The initial vectors described above are packaged into two sets of trainable fine-grained text parameter matrices:

[0012] And use the initial text features for initialization. During the subsequent training process, the matrix and As a learnable parameter, it will be dynamically updated based on image feedback, gradually absorbing fine-grained discrimination information from the visual modality; Fine-grained branches of combined text are obtained by introducing a fine-grained fusion mechanism for attribute object text; The model performs parallel computation along two paths; Enhance the fine-grained nature of image-guided text.

[0013] Optionally, step three includes: The multi-scale semantic fine-grained embedding and prediction output unit implements differentiated residual enhancement based on the essential differences in visual information requirements among the three semantic levels of attributes, objects, and combinations; the attribute branch focuses on local high-fine-grained visual features and uses high-weight gating for strong enhancement. The object branch focuses more on structural robustness, applying only low-weight gating to suppress image noise interference; the combination branch introduces learnable adaptive gating parameters to dynamically balance the fusion ratio of combined text prototypes and combined semantic fine-grainedness. Finally, the enhanced multi-scale text features are aligned with image branch features for prediction, improving the model's accuracy and generalization ability in discriminating unseen combinations.

[0014] Optionally, the step of obtaining the combined text fine-grained branches by introducing an attribute object text fine-grained fusion mechanism includes:

[0015] in ( ) introduces a residual connecter with learnable coefficients based on the combined features generated by linear stitching projection for fine-grained attributes and objects.

[0016] Optionally, the steps of the model to compute the two paths in parallel include: the first path concatenates the attributes and object features and passes them through a fully connected layer to obtain basic combined features; the second path adds the attributes and object features element by element to obtain residual terms containing attribute-object interaction information; subsequently, the residual terms are multiplied by a learnable scalar coefficient and added back to the basic combined features to form the final fine-grained output of the combined text.

[0017] Optionally, the step of enhancing the image-guided text at a fine-grained level includes: employing an image-guided cross-attention module to, for each sample, apply image features... As a query, three fine-grained text features Use them as keys and values ​​respectively, and perform cross-attention calculations separately:

[0018] in Standard multi-head attention; thus obtained , , These are fine-grained text features guided by images, which retain the original text semantics while incorporating the visual context of the current image; Finally, , , The data is stitched along the channels and fused through a linear projection layer to form the final image-guided fine-grained text features, which are then used for subsequent multi-scale gating enhancement.

[0019] A zero-shot learning system based on multi-scale semantic fine-grained embedding, used to execute the aforementioned zero-shot learning method based on multi-scale semantic fine-grained embedding, includes: The system includes an image input unit, a text input unit, a CLIP feature encoding unit, a traditional text prototype three-branch module, an image-guided fine-grained text extraction module, and a multi-scale semantic fine-grained embedding and prediction output unit.

[0020] The present invention has the following beneficial effects: This invention can inject fine-grained semantic information trained by image guidance into the text learning process, and design differentiated enhancement strategies for different semantic levels to achieve high-precision recognition of unseen combinations.

[0021] By improving upon the traditional three-path approach of combinatorial zero-shot learning through learnable fine-grained text matrices, image-guided fine-grained semantic enhancement, and multi-scale fine-grained semantic embedding, the model effectively integrates visual discriminative information while preserving textual semantic priors. This significantly improves the feature alignment accuracy at the three semantic levels of attributes, objects, and combinations, and reduces modality gap. Ultimately, the model can achieve higher accuracy in recognizing unseen attribute-object combinations, fully demonstrating the technical advantages of this invention. Attached Figure Description

[0022] The present invention will be further described with reference to the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the present invention. For those skilled in the art, other drawings can be obtained based on the following drawings without creative effort.

[0023] Figure 1 This is a flowchart of the zero-shot learning method based on multi-scale semantic fine-grained embedding according to the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] In the description of this invention, it should be noted that the terms "vertical," "upper," "lower," and "horizontal," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, "first," "second," "third," and "fourth" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0026] In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or a connection through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0027] like Figure 1 As shown, a zero-shot learning method based on multi-scale semantic fine-grained embedding includes the following steps: Step 1: Process the tooltip text of attributes, objects, and combinations using the frozen CLIP text encoder; Step 2: Perform fine-grained initial extraction of the text; Step 3: Enhance the semantic hierarchy modeling capability in the traditional text prototype three-branch architecture by leveraging multi-scale semantic fine-grained embedding and predictive output units.

[0028] In an optional embodiment of the present invention, step 1 includes the following sub-steps: Step 1.1: The traditional text prototype three-branch module constructs three types of learnable hints: attributes, objects, and combinations. These hints are then encoded by the CLIP text encoder to obtain the corresponding level of text prototype features.

[0029] In an optional embodiment of the present invention, step 1 includes the following sub-steps: Step 1.2: On the image side, the visual features are decomposed into independent features of attribute features and object features through the multi-head cross-attention decoupling module, so that the visual representation of attributes, objects and combinations is accurately aligned with the text prototype representation, thus solving the problem of entanglement between attributes and objects in visual features. During training, cross-modal attention scores between image and text prototypes are incorporated into similarity calculations to optimize the prediction probabilities of attributes and object branches.

[0030] In an optional embodiment of the present invention, step 1 includes the following sub-steps: Step 1.3: Train the module using cross-entropy loss and update the parameters of the hint and decoupling modules.

[0031] The traditional three-branch text prototype module provides a stable semantic classification benchmark for combinatorial zero-shot learning, serving as a stable foundation for subsequent multi-scale semantic fine-grained embedding and constraining semantic fine-grained extraction to not deviate from the original information.

[0032] In an optional embodiment of the present invention, step two includes: In addition to the traditional three branches of the text prototype, three fine-grained branches for text prototype enhancement are introduced. The image-guided fine-grained text extraction module is responsible for extracting features from the input image to provide an information basis for subsequent semantic fine-grained learning and text prototype fine-grained enhancement. The initial attribute and object text fine-grained branches are the same as the attribute object branches of the text prototype. The frozen CLIP text encoder is used to process the cue text of attributes, objects and combinations, which serves as the basis for learning the subsequent image-guided text fine-grained extraction module. Unlike attribute and object text fine-grained branches, which are encoded by the encoder, the combined text fine-grained branches are obtained by using the initial attribute and text fine-grained branches through the initial attribute and object text fine-grained fusion mechanism to obtain the initial combined text fine-grained branches. The branches of fine-grained attribute and object text are initialized from the same source as the text prototype, while the combined fine-grained text is dynamically generated from the initial fine-grained text of attributes and objects through an innovative fusion mechanism; the process is as follows: First, for each attribute word in attribute set A (i represents the attribute word index, 1, 2, 3...) and each object word in the object set O. (j represents the word index, 1, 2, 3, ...), using a pre-trained CLIP text encoder. ( ) Extract its initial text fine-grained vector:

[0033] Where d is the feature dimension; Unlike traditional methods that treat it as a fixed prototype, this invention packages the above initial vector into two sets of trainable fine-grained text parameter matrices:

[0034] And use the initial text features for initialization. ; This initialization strategy utilizes the rich semantic priors contained in large-scale pre-trained language models as a foundation, enabling fine-grained branches to have a reasonable semantic distribution at the beginning of training, effectively avoiding the problems of training instability and slow convergence caused by random initialization. During the subsequent training process, the matrix and As a learnable parameter, it will be dynamically updated based on image feedback, gradually absorbing fine-grained discrimination information from the visual modality; If the fine-grained features of combined text are generated by MLP after concatenating attribute and object features, the interaction information between the two is easily lost. Therefore, this invention obtains the fine-grained branch of combined text by introducing an attribute-object text fine-grained fusion mechanism. The model performs parallel computation along two paths; Fine-grained enhancement of image-guided text. In an optional embodiment of the invention, step three includes: The multi-scale semantic fine-grained embedding and prediction output unit implements differentiated residual enhancement based on the essential differences in visual information requirements among the three semantic levels of attributes, objects, and combinations; the attribute branch focuses on local high-fine-grained visual features and uses high-weight gating for strong enhancement. The object branch focuses more on structural robustness, applying only low-weight gating to suppress image noise interference; the combination branch introduces learnable adaptive gating parameters to dynamically balance the fusion ratio of combined text prototypes and combined semantic fine-grainedness. Finally, the enhanced multi-scale text features are aligned with image branch features for prediction, improving the model's accuracy and generalization ability in discriminating unseen combinations.

[0035] In an optional embodiment of the present invention, the step of obtaining combined text fine-grained branches by introducing an attribute object text fine-grained fusion mechanism includes:

[0036] in ( ) introduces a residual connecter with learnable coefficients based on the combined features generated by linear stitching projection for fine-grained attributes and objects.

[0037] In an optional embodiment of the present invention, the step of the model computing two paths in parallel includes: the first path concatenates the attributes and object features and passes them through a fully connected layer to obtain basic combined features; the second path adds the attributes and object features element by element to obtain residual terms containing attribute-object interaction information; subsequently, the residual terms are multiplied by a learnable scalar coefficient and added back to the basic combined features to form the final fine-grained output of combined text.

[0038] To enhance the visual discriminative power of fine-grained text features, in an optional embodiment of the present invention, the step of enhancing the fine-grained image-guided text features includes: employing an image-guided cross-attention module to, for each sample, apply image features... As a query, three fine-grained text features Use them as keys and values ​​respectively, and perform cross-attention calculations separately:

[0039] in Standard multi-head attention; thus obtained , , These are fine-grained text features guided by images, which retain the original text semantics while incorporating the visual context of the current image; Finally, , , The data is stitched along the channels and fused through a linear projection layer to form the final image-guided fine-grained text features, which are then used for subsequent multi-scale gating enhancement.

[0040] During training, the image-guided cross-attention module uses a contrastive learning approach to bring image features closer to corresponding fine-grained text features and further away from non-corresponding fine-grained text features, thereby improving the ability to distinguish between similar combinations. A zero-shot learning system based on multi-scale semantic fine-grained embedding, used to execute the aforementioned zero-shot learning method based on multi-scale semantic fine-grained embedding, includes: The system includes an image input unit, a text input unit, a CLIP feature encoding unit, a traditional text prototype three-branch module, an image-guided fine-grained text extraction module, and a multi-scale semantic fine-grained embedding and prediction output unit. Example

[0041] In this embodiment, taking an input image and its corresponding attribute-object combined text as an example, the specific implementation process of the multi-scale semantic fine-grained embedding proposed in this invention is illustrated. First, in the preprocessing input stage, the input image is scaled to a fixed size specified by pre-training (e.g., 224×224 pixels) after preprocessing, and then divided into 16×16 image blocks by the convolutional layer of the visual encoder, resulting in 256 image block vectors. A global category token is added at the beginning, thus forming 257 image input vectors, each with a dimension of 768. Simultaneously, the text part is constructed using preset prompt templates, such as "a photo of {attr} {obj}" for the combined path, "a photo of {attr} object" for the attribute path, and "a photo of {obj}" for the object path. Here, "{attr}" corresponds to attribute words (e.g., "red"), and "{obj}" corresponds to object words (e.g., "truck"), all derived from a preset learnable attribute-object vocabulary. After processing by the text encoder, the text is embedded into text for the combined, attribute, and object paths, with each embedding having a basic dimension of 768. In the feature encoding stage, the image input is processed by a frozen image encoder for feature extraction. Simultaneously, a learnable adapter module is inserted after the attention module and MLP (Multilayer Perceptron) module in each layer of the encoder for lightweight adaptation, ultimately outputting a global image feature representation (composed of global and local feature representations, with a shape of (257, 768)). The image features are then processed by two independent deentanglement modules to obtain image attribute features and image object features, which are superimposed on the original global image combination features to form the features of the three visual paths: image combination features, image attribute features, and image object features. On the other end, the text input is processed by a frozen text encoder to encode the embeddings of the combination, attribute, and object paths respectively, resulting in combination text prototype features, attribute text prototype features, and object text prototype features, each with a dimension of 768. These three constitute the semantic baseline of the traditional three-branch architecture. Furthermore, this embodiment also pre-calculates a learnable fine-grained feature matrix by averaging the word embeddings of the attribute vocabulary and object vocabulary using the CLIP text encoder. (Shape is |A|×768, where |A| is the total number of attribute words) and the fine-grained feature matrix of the object. (Shape is |O|×768, where |O| is the total number of object words), serving as the basis for fine-grained semantic extraction; each attribute word is extracted using a frozen CLIP text encoder. and object words Embedded vector As initial values, these parameters are dynamically updated during training based on image feedback, gradually absorbing fine-grained discriminative information from visual modalities; Dynamic fusion of fine-grained text features: combining fine-grained features It is not directly encoded by CLIP, but rather by fine-grained features of attributes. fine-grained features of objects Generated through innovative integration mechanisms:

[0042] The fusion mechanism comprises two paths: the first path involves projecting the concatenated data through a fully connected layer, and the second path involves element-wise addition of residual connections multiplied by learnable scalar coefficients. ; With 768 dimensions, this design retains the interaction information between attributes and objects, while enhancing the ability to generalize to unseen compositions; The second part is image-guided fine-grained text enhancement; to inject the visual context of the current image, this invention uses a cross-attention mechanism to perform image-guided enhancement on three types of fine-grained text features; specifically, global image features... As a query, fine-grained text features are used as keys and values ​​respectively:

[0043] in ( ) represents the normalization of the standard multi-head attention follow-up layer; the resulting It preserves the original text semantics while incorporating image-guided discriminative information; then, the three are concatenated along the channel and linearly projected to obtain the overall fine-grained features; After image-guided fine-grained enhancement, multi-scale semantic fine-grained embedding is then performed: differential residual enhancement is implemented based on the essential differences in visual information requirements among the three semantic levels of attributes, objects, and combinations; attribute branch: the input consists of attribute text prototype features and image-guided attribute fine-grained features. By using a high-weight learnable gating parameter with an initial value of 0.8, Superimposed on text prototype features, this enhances the ability to capture attribute semantics, outputting enhanced attribute text features with 768 dimensions; Object branch: Input is object text prototype features and image-guided fine-grained object features. By using a low-weight learning gate parameter with an initial value of 0.2, the gating parameters can be... Superimposed on text prototype features, balancing the semantic contributions of attributes and objects, the output is an enhanced object text feature with 768 dimensions; Combination branch: the input is the combined text prototype feature and... Feature tuples, through softmax normalization of 2D learnable gating parameters to obtain adaptive weights, are then weighted... By overlaying the prototype features, a dynamic balance between attributes and object semantics is achieved, solving the problem of entanglement of combined semantics, and outputting enhanced combined text features with a dimension of 768. All enhanced text features are normalized and matched with the image feature dimensions for subsequent similarity calculation. In the similarity calculation stage, cosine similarity is calculated for three groups: the similarity between visual attributes and text attributes, the similarity between visual objects and text objects, and the similarity between visual combinations and text combinations. Let the weights for these three be α, β, and γ, respectively. Then, the final combination score is:

[0044] In the formula: S is the similarity score (final score) of the image as a whole of a certain combination; α, β, and γ are three weighting coefficients (hyperparameters) that control the contribution ratio of attributes, objects, and combinations, respectively; sim is the similarity function; , , These are image attribute features, image object features, and image combination feature representations, respectively. , , These are feature representations of attributes, objects, and combinations of text after multi-scale fine-grained text enhancement; By calculating the score between the input image and the candidate combination label set, the label with the highest score is selected as the prediction result; for example, when the input is an image of a red truck, the system can correctly output the combination label "red truck" even if this combination has never been seen during training. During training, the process is the same as inference, but a supervision signal is added. By comparing the loss function, the similarity of correct image-text pairs is required to be as high as possible, and the similarity of incorrect image-text pairs is required to be as low as possible. The loss function is composed of the alignment loss of the attribute layer, object layer, and combination layer, thereby constraining the feature alignment effect at multiple levels simultaneously. The parameters updated during training include the fine-grained feature matrices of attributes and objects, the cross-attention module, the multi-scale gating enhancement module, and the mapping parameters inside the alignment and decoupling module, while the image and text backbone encoders are kept frozen to ensure training stability and reduce computational overhead. This embodiment fully illustrates the forward processing flow of a combined zero-shot learning method based on multi-scale semantic fine-grained embedding. Through learnable text fine-grained matrix initialization, image-guided cross-attention enhancement, and multi-scale differential gating embedding, the model effectively integrates visual discrimination information while preserving text semantic priors, significantly improving the feature alignment accuracy of the three semantic levels of attributes, objects, and combinations. Finally, the model can accurately identify unseen attribute-object combinations, fully demonstrating the technical advantages of this invention.

[0045] This invention can inject fine-grained semantic information trained by image guidance into the text learning process, and design differentiated enhancement strategies for different semantic levels to achieve high-precision recognition of unseen combinations.

[0046] By improving upon the traditional three-path approach of combinatorial zero-shot learning through learnable fine-grained text matrices, image-guided fine-grained semantic enhancement, and multi-scale fine-grained semantic embedding, the model effectively integrates visual discriminative information while preserving textual semantic priors. This significantly improves the feature alignment accuracy at the three semantic levels of attributes, objects, and combinations, and reduces modality gap. Ultimately, the model can achieve higher accuracy in recognizing unseen attribute-object combinations, fully demonstrating the technical advantages of this invention.

[0047] The components, modules, mechanisms, and devices in this invention that are not described in detail are all general standard parts or components known to those skilled in the art. Their structures and principles can be learned by those skilled in the art through technical manuals or conventional experimental methods.

[0048] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit the scope of protection of the present invention. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the essence and scope of the technical solutions of the present invention.

Claims

1. A zero-shot learning method based on multi-scale semantic fine-grained embedding, characterized in that, Includes the following steps: Step 1: Process the tooltip text of attributes, objects, and combinations using the frozen CLIP text encoder; Step 2: Perform fine-grained initial extraction of the text; Step 3: Enhance the semantic hierarchy modeling capability in the traditional text prototype three-branch architecture by leveraging multi-scale semantic fine-grained embedding and predictive output units.

2. The zero-shot learning method based on multi-scale semantic fine-grained embedding according to claim 1, characterized in that, Step 1 includes the following sub-steps: Step 1.1: The traditional text prototype three-branch module constructs three types of learnable hints: attributes, objects, and combinations. These hints are then encoded by the CLIP text encoder to obtain the corresponding level of text prototype features.

3. The zero-shot learning method based on multi-scale semantic fine-grained embedding according to claim 2, characterized in that, Step 1 includes the following sub-steps: Step 1.2: On the image side, the visual features are decomposed into independent features of attribute features and object features through the multi-head cross-attention decoupling module, so that the visual representation of attributes, objects and combinations is accurately aligned with the text prototype representation, thus solving the problem of entanglement between attributes and objects in visual features. During training, cross-modal attention scores between image and text prototypes are incorporated into similarity calculations to optimize the prediction probabilities of attributes and object branches.

4. The zero-shot learning method based on multi-scale semantic fine-grained embedding according to claim 3, characterized in that, Step 1 includes the following sub-steps: Step 1.3: Train the module using cross-entropy loss and update the parameters of the hint and decoupling modules.

5. The zero-shot learning method based on multi-scale semantic fine-grained embedding according to claim 4, characterized in that, Step two includes: In addition to the traditional three branches of the text prototype, three fine-grained branches for text prototype enhancement are introduced. The image-guided fine-grained text extraction module is responsible for extracting features from the input image to provide an information basis for subsequent semantic fine-grained learning and text prototype fine-grained enhancement. The initial attribute and object text fine-grained branches are the same as the attribute object branches of the text prototype. The frozen CLIP text encoder is used to process the cue text of attributes, objects and combinations, which serves as the basis for learning the subsequent image-guided text fine-grained extraction module. Unlike attribute and object text fine-grained branches, which are encoded by the encoder, the combined text fine-grained branches are obtained by using the initial attribute and text fine-grained branches through the initial attribute and object text fine-grained fusion mechanism to obtain the initial combined text fine-grained branches. The branches of fine-grained attribute and object text are initialized from the same source as the text prototype, while the combined fine-grained text is dynamically generated from the initial fine-grained text of attributes and objects through an innovative fusion mechanism; the process is as follows: First, for each attribute word in attribute set A (i represents the attribute word index, 1, 2, 3...) and each object word in the object set O. (j represents the word index, 1, 2, 3, ...), using a pre-trained CLIP text encoder. ( ) Extract its initial text fine-grained vector: ； Where d is the feature dimension; The initial vectors described above are packaged into two sets of trainable fine-grained text parameter matrices: ； And use the initial text features for initialization. During the subsequent training process, the matrix and As learnable parameters, they will be dynamically updated based on image feedback, gradually absorbing fine-grained discrimination information from the visual modality; Fine-grained branches of combined text are obtained by introducing a fine-grained fusion mechanism for attribute object text; The model performs parallel computation along two paths; Enhance the fine-grained nature of image-guided text.

6. The zero-shot learning method based on multi-scale semantic fine-grained embedding according to claim 5, characterized in that, Step three includes: The multi-scale semantic fine-grained embedding and prediction output unit implements differentiated residual enhancement based on the essential differences in visual information requirements among the three semantic levels of attributes, objects, and combinations; the attribute branch focuses on local high-fine-grained visual features and uses high-weight gating for strong enhancement. The object branch focuses more on structural robustness, applying only low-weight gating to suppress image noise interference; the combination branch introduces learnable adaptive gating parameters to dynamically balance the fusion ratio of combined text prototypes and combined semantic fine-grainedness. Finally, the enhanced multi-scale text features are aligned with image branch features for prediction, improving the model's accuracy and generalization ability in discriminating unseen combinations.

7. The zero-shot learning method based on multi-scale semantic fine-grained embedding according to claim 6, characterized in that, The steps for obtaining combined text fine-grained branches by introducing an attribute object text fine-grained fusion mechanism include: ； in ( ) introduces a residual connecter with learnable coefficients based on the combined features generated by linear stitching projection for fine-grained attributes and objects.

8. The zero-shot learning method based on multi-scale semantic fine-grained embedding according to claim 7, characterized in that, The model's parallel computation of two paths includes: the first path concatenates attributes and object features and passes them through a fully connected layer to obtain basic combined features; the second path adds attributes and object features element by element to obtain residual terms containing attribute-object interaction information; subsequently, the residual terms are multiplied by a learnable scalar coefficient and added back to the basic combined features to form the final fine-grained output of combined text.

9. A zero-shot learning method based on multi-scale semantic fine-grained embedding according to claim 8, characterized in that, The step of enhancing the image-guided text at a fine-grained level includes: employing an image-guided cross-attention module to, for each sample, apply image features... As a query, three fine-grained text features Use them as keys and values ​​respectively, and perform cross-attention calculations separately: ； in This is standard multi-head attention; the result is... , , These are fine-grained text features guided by images, which retain the original text semantics while incorporating the visual context of the current image; Finally, , , The data is stitched along the channels and fused through a linear projection layer to form the final image-guided fine-grained text features, which are then used for subsequent multi-scale gating enhancement.

10. A zero-shot learning system based on multi-scale semantic fine-grained embedding, characterized in that, A zero-shot learning method based on multi-scale semantic fine-grained embedding as described in any one of claims 1-9, comprising: The system includes an image input unit, a text input unit, a CLIP feature encoding unit, a traditional text prototype three-branch module, an image-guided fine-grained text extraction module, and a multi-scale semantic fine-grained embedding and prediction output unit.

Images