Generative zero-shot learning method based on dual-state collaborative decoupling and semantic refinement

Abstract

The present invention discloses a generative zero-shot learning method based on dual-state collaborative decoupling and semantic refinement, which belongs to the field of generative zero-shot learning. This method realizes the complementary expression of static and dynamic features by decoupling the static features such as background, structure and details of the image, and extracting dynamic common features by a cross-modal label generation module; dynamically adjusts the focus of features according to different confusion types, enhances feature discriminability, and alleviates cross-category interference; at the semantic level, by constructing a visual-semantic mirror cross-attention mechanism, it realizes bidirectional alignment between semantic features and visual features, further improving the multi-granularity ability and adaptability of semantic representation. The present invention takes feature structure decoupling, confusion adaptive regulation and dynamic semantic alignment as the core, effectively improving the cross-category generalization ability and the quality of generated samples, breaking through the performance bottleneck of traditional generative zero-shot learning, and has strong theoretical value and broad application prospects.

Description

Technical Field

[0001] The present invention relates to the field of generative zero-shot learning, and in particular to a generative zero-shot learning method based on dual-state collaborative decoupling and semantic refinement. Background Art

[0002] Zero-shot learning (ZSL) aims to recognize unseen categories from semantic descriptions (such as attribute vectors or text embeddings). In recent years, both embedded and generative methods have made significant progress in addressing this challenge. Embedded methods attempt to map visual features into a shared semantic space, while generative methods such as Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs) generate visual features for unseen categories from given semantic descriptions such as attribute vectors or text embeddings, thereby reducing the reliance on real labeled samples.

[0003] Traditional semantic mapping methods struggle with the “seen-unseen” category bias problem as they often fail to adequately distinguish between seen and unseen categories, especially when their semantic descriptions overlap. These methods typically focus on visual-semantic alignment but still face issues with the discriminative power of generated features. Furthermore, the use of predefined static semantic prototypes limits the generalization ability of the models as they fail to exploit the dynamic relationships between attributes. To overcome the limitations of traditional methods, generative ZSL methods have attracted widespread attention. These methods use deep generative models such as GANs and VAEs to generate features for unseen categories, thereby alleviating the data scarcity problem by creating synthetic data for unseen categories. Furthermore, recent advances have introduced more sophisticated techniques such as dual alignment methods and prototype augmentation to refine feature generation. These innovations aim to reduce domain bias and improve the discriminative power of synthesized features.

[0004] Current generative zero-shot learning methods face several key bottlenecks: First, the visual features of images, such as background, posture, and texture, are not effectively decoupled, resulting in severe coupling between static features, which in turn exacerbates confusion between individuals of the same type and the shift of cross-class features, limiting the expressive power of cross-modal features; second, existing methods lack effective reuse of the diversity of features of seen classes in the generation of unseen class samples, and fail to model combinatorial associations such as "background-structure", resulting in insufficient typicality and authenticity of generated samples and prone to domain shift; third, traditional generators mostly rely on fixed semantics as input and lack a dynamic feedback mechanism based on training status, making it difficult to effectively improve the alignment accuracy and diversity of generated samples. Summary of the Invention

[0005] Purpose of the invention: In response to the above problems, the purpose of the present invention is to provide a generative zero-shot learning method based on dual-state collaborative decoupling and semantic refinement, with feature structure decoupling, confusion adaptive regulation and dynamic semantic alignment as the core, which effectively improves the cross-category generalization ability and the quality of generated samples, breaking through the performance bottleneck of traditional generative ZSL.

[0006] Technical Solution: On one hand, the present invention provides a generative zero-shot learning method based on dual-state collaborative decoupling and semantic refinement, comprising the following steps:

[0007] Step 1: Input the target modal data into the feature extraction module to extract the global modal features; input the global modal features into the static decoupling module to perform multi-level static decoupling and separation on the features to obtain static features of different levels;

[0008] Step 2: The global modal features and static features at different levels are input into the dynamic decoupling module. The channel attention mechanism is used to filter and extract the shared dynamic patterns in the modal features.

[0009] Step 3: Input static features and dynamic patterns at different levels into the prototype memory module to generate reusable feature combinations and provide initial feature prototypes for unseen classes. During training, the feature weights are dynamically adjusted based on the feedback from the confusion detection module, and a conditional vector is generated to fit Gaussian noise. The conditional vector is then input into the generative model.

[0010] Step 4: Input the dynamic pattern into the semantic refinement module, generate corresponding semantic features through the mirrored cross-attention mechanism, and align the global modal features with the semantic features to obtain the refined semantic representation;

[0011] Step 5: Using the refined semantic representation and conditional vector, the generative model synthesizes samples of seen and unseen classes, and inputs them into the zero-shot classifier to complete comprehensive reasoning. During the reasoning process, based on the feedback from the classifier or discriminator, the same-class and cross-class confusion are identified, and the features in the prototype memory are updated through the confusion detection module. Finally, the updated semantic representation is input into the generative model.

[0012] Step 6: Combine the discriminator, classifier, and feature matching error indicators and use the semantic feedback module to dynamically evaluate the quality of the generated samples. Based on the evaluation results, the dynamic semantic refinement module is optimized to adjust the weights of each feature level.

[0013] Step 7: Repeat steps 3 to 6 and calculate the top-1 until the iteration termination condition is met to obtain the final neural network architecture.

[0014] Step 8: During the testing phase, the synthesized unseen class samples are used to enhance the generalization ability of the classifier, ultimately achieving the goal of zero-shot learning and accurately identifying the modal data of seen and unseen classes.

[0015] Furthermore, step 1 includes:

[0016] The target modality data is visual modality or audio modality; for the visual modality, the global visual features are recorded as , the global visual features Converting to image-labeled sequences via a visual transformer , and add learnable category labels , forming a complete input feature ;

[0017] For the audio modality, the audio is converted into a Mel spectrum, the time-frequency features of the Mel spectrum are enhanced through the fine-tuning module, and the result is input into the visual Transformer to convert the audio modality into the visual modality.

[0018] Compute input features through query, key, and value mapping functions The attention weight , the formula is: , where For the The query vector of the position is expressed as , is the input feature matrix The i-th row of N is the number of positions, d is the feature dimension, represents a learnable mapping function; represents the bond matrix, represents transpose, is the scaling factor, a is the head number;

[0019] The value vector V of all spatial positions is calculated according to the corresponding attention weight Perform linear weighting and then aggregate along the spatial dimension to generate tuned features , Represents element-by-element multiplication; and upsampling restores the spatial dimension to obtain features , Indicates height, W indicates width, and C indicates the number of channels;

[0020] Through multi-level feature decomposition encoding, the features Decomposed into three levels of static features: background level features, structure level features and detail level features;

[0021] Among them, the expression of background level features is:

[0022] ,

[0023] Where, represents channel attention, represents a 3×3 depthwise separable convolution, Represents 1×1 convolution;

[0024] The expression of the structural level feature is:

[0025] ,

[0026] Where, represents spatial attention, Represents 3×3 dilated convolution;

[0027] The expression of detail level feature is:

[0028] ,

[0029] Where, represents local self-attention, represents the residual block;

[0030] The background-level features and structure-level features are fused through cross-attention to update the background-level features. The formula is:

[0031] ,

[0032] Where, is the updated background-level feature, represents the cross attention mechanism;

[0033] The detail-level features are dynamically combined with the structure-level features through gated weighting to update the structure-level features. The formula is:

[0034] ,

[0035] Where, represents the updated structural features, Represents the gating coefficient, and the calculation formula is:

[0036] ,

[0037] Where, Represents the Sigmoid activation function.

[0038] Furthermore, step 2 includes:

[0039] Global visual features Combined with multi-level static features into joint features , the formula is:

[0040] ,

[0041] Filtering joint features through spatial-channel attention The high-frequency activation area generates common features, and the formula is:

[0042] ,

[0043] ,

[0044] Where, express activation function, is the attention mask, Represents common characteristics, Represents a 3×3 convolution operation;

[0045] Decompose common features into two types of dynamic visual features as dynamic patterns: structural dynamic features and detail dynamic features;

[0046] The expression of the structural dynamic characteristics is:

[0047] ,

[0048] The expression of the detailed dynamic characteristics is:

[0049] ,

[0050] The output structural dynamic features and detail dynamic features are used as pseudo-label representations of the image.

[0051] Furthermore, in step 2, when decomposing the common features into two types of dynamic visual features, a new triplet loss function is constructed. The process includes:

[0052] Perform cross-sample mean pooling on dynamic features to generate common prototypes. The formula is:

[0053] ,

[0054] Where, represents the dynamic feature vector corresponding to the i-th sample, and N represents the total number of dynamic feature vectors participating in mean pooling;

[0055] Common prototype As an anchor , taking the personality characteristics of different instances of the same category as positive samples , among which personality traits Use static features directly or , personality traits of other categories are used as negative samples ;

[0056] Construct the loss function, the expression is:

[0057] , where m represents the interval hyperparameter in the loss, which is used to control the minimum separation between positive and negative samples and anchor points.

[0058] Furthermore, step 3 includes:

[0059] The structural level features, detail level features, and background level features are directly used as the personality prototypes within the class, without participating in cross-class aggregation, and a static weight matrix is ​​generated through offline pre-training. , fixed static weight matrix after training , the formula is:

[0060] ,

[0061] Where, represents a multilayer perceptron with two linear layers, Represents the static prototype vector of the cth class, the formula is:

[0062] ,

[0063] Where, represents the number of samples in the cth class, represents the structural static features of the i-th sample of the c-th category, represents the detail-level static features of the i-th sample of the c-th category;

[0064] Perform cross-sample mean pooling on the structural dynamic features and detail dynamic features to generate dynamic commonality prototypes. The formula is:

[0065] ,

[0066] Where, represents the structural dynamic characteristics of the i-th sample of the c-th category, Represents the detailed dynamic features of the i-th sample of the c-th category;

[0067] Update the generated data prototype using the dynamic features output by the generated model With static features Update prototypes separately , , where represents the intra-class mean of the dynamic features output by the generated model at the t-th iteration, Represents the intra-class mean of the static features output by the generated model at the t-th iteration;

[0068] The prototype alignment is then performed through a dynamic update mechanism, where Force the dynamic characteristics of the generated model to approach the real dynamic prototype, by Perform static feature alignment; where Represents a static prototype vector;

[0069] At the completion of training, the static features are noise-fitted, that is, the static prototypes and feature weights are combined into learnable distribution parameters, including the mean and variance , the formulas are:

[0070] ,

[0071] ,

[0072] Where, It means that it is calculated through a linear layer. represents the static characteristics of the cth class, , the joint posterior distribution is explicitly defined in terms of mean and variance, as follows:

[0073] ,

[0074] Introducing KL divergence constraint loss, the formula is:

[0075] ,

[0076] Where tr is the trace operation of the matrix, d is the dimension of the latent variable, and log det is the logarithm of the determinant;

[0077] Mix the original Gaussian noise to generate the conditional vector , the formula is:

[0078] ,

[0079] Where, is the weight, which controls the proportion of original noise retained. and are all represented as Gaussian noise.

[0080] Furthermore, step 4 includes:

[0081] The weighted static features and dynamic features in the static decoupling module are spliced ​​into joint visual features. The formula is:

[0082] ,

[0083] The joint visual features are converted into initial semantic features through a fixed parameter mapping network R ;

[0084] The joint visual features are used as the query value in the attention mechanism, the initial semantic features are used as the key value and the true value, and the multi-granularity information in the visual features is captured through multi-head cross attention to obtain the enhanced semantic features. The formula is:

[0085] ,

[0086] Where, represents the query vector, represents the key vector, represents a value vector, represents the scaling factor;

[0087] The enhanced semantic features are then used to regenerate visual features through a fixed parameter generation model ;

[0088] Semantic-visual cross attention is introduced, and the enhanced semantic features are used as query values. The newly generated visual features are used as key values ​​and true values. Multi-head cross attention is used to strengthen the multi-granularity connections in the semantic features, and the enhanced visual features are obtained. The formula is:

[0089] ,

[0090] Where, 、 Represent the query vector, key vector and value vector respectively.

[0091] Furthermore, step 5 includes:

[0092] First, the confidence and entropy values ​​of the classifier output are analyzed in real time. When the entropy value of the prediction result is greater than the threshold and fluctuates repeatedly between similar instances, it is judged as similar confusion. The loss makes the dynamic features of similar samples similar, and the static features and dynamic features of similar samples complementary; The loss formula is:

[0093] ,

[0094] Where, , ;

[0095] Among them, when the classification result mistakenly falls into a category with significantly different semantics, it is judged as cross-class confusion, and the The loss increases the dynamic feature distance between samples of different classes, and the formula is:

[0096] ,

[0097] Where, is a predefined cross-class confusion pair, is the interval hyperparameter;

[0098] Utilize dynamic gating mechanism Adaptively generate the attention weights of each feature level, the formula is: , where f uniformly represents the feature tensor extracted at different semantic levels.

[0099] Furthermore, in step 6, dynamic evaluation uses a multi-layer perceptron, which takes the discriminator confidence, classifier entropy and feature matching error as input and outputs a dynamic threshold and loss weight adjustment coefficient: ,in, is the penalty coefficient for the dynamic threshold growth of the hyperparameter, The dynamic threshold is used to determine whether to trigger semantic feature updates. When the discriminator confidence is too low, the classifier entropy is too high, or the feature matching error exceeds the preset threshold, the module automatically starts optimizing the semantic features.

[0100] After triggering the semantic update, the module optimizes the cross-attention module parameters in the semantic refinement module through gradient backpropagation; at the same time, the loss weight adjustment coefficient is used to adaptively adjust the weight distribution of each loss item in the semantic refinement module. , thereby dynamically optimizing the semantic refinement process.

[0101] Another aspect of the present invention provides a generative zero-shot learning system based on dual-state collaborative decoupling and semantic refinement, the system comprising:

[0102] The static decoupling module tunes the global mode through preprocessing operations; the feature decomposition encoder further performs three-level static decoupling of the tuned features at the background level, structure level, and detail level;

[0103] Dynamic decoupling module, used to further extract common patterns from global modal features and static features to achieve dual-state collaborative decoupling of static and dynamic features;

[0104] The prototype memory module is used to dynamically update static and dynamic features. By dynamically adjusting feature weights, it generates a conditional vector that fits Gaussian noise and provides guidance for feature generation.

[0105] The semantic refinement module uses mirror-symmetric cross-attention to achieve bidirectional refinement and update of visual and semantic features to obtain refined semantic representations;

[0106] The confusion detection module analyzes the same-class and cross-class confusion based on the feedback of the classifier prediction, and generates and adjusts the attention weights of each feature level in real time through a dynamic gating mechanism;

[0107] The semantic feedback module automatically triggers the semantic update timing of the semantic refinement module based on the generation quality by comprehensively evaluating the discriminator confidence, classifier entropy and feature matching error.

[0108] Furthermore, the system includes alignment loss, mirror symmetry loss, distribution consistency loss and feature matching loss, and dynamically adjusts the weights of each loss item according to the semantic feedback module, optimizes the parameters of each module through back propagation, and iterates until the generation quality reaches a preset threshold; finally, high-quality generated visual features are output.

[0109] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0110] 1. The static decoupling module of the present invention can effectively filter redundant noise and efficiently decouple the original modal features at multiple levels, providing structured representation for subsequent feature processing and alleviating the serious coupling problem of static features in traditional methods.

[0111] 2. Through the dynamic decoupling module, we propose to extract cross-category dynamic features from static features, which enhances the inter-class discrimination ability and intra-class generalization ability of dynamic features;

[0112] 3. By maintaining real data and generated data prototypes through the prototype memory module, we successfully achieved effective management and adaptive updating of static and dynamic features, and further alleviated the domain shift problem by distinguishing between same-class confusion and cross-class confusion;

[0113] 4. The semantic refinement module innovatively adopts a mirror-symmetric cross-attention mechanism to achieve bidirectional alignment and dynamic updating of visual and semantic features, overcoming the problem of insufficient static semantic expression in traditional methods.

[0114] 5. The confusion detection module effectively solves the problem of feature confusion in zero-shot learning tasks. By analyzing the entropy and confidence of the classifier's predictions in real time, dynamically determining the type of confusion, and adjusting the attention weights at the feature level in real time, the model's discriminative ability is significantly enhanced, effectively reducing misclassification within the same category and across categories.

[0115] 6. By designing a method for binary decoupling of features and semantic refinement, the discriminability and generalization ability of the generated features of the zero-shot learning model are significantly improved, effectively solving key problems such as semantic solidification, lack of generated features, and domain shift in traditional generative zero-shot learning tasks. It has significant technical advantages and broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0116] Figure 1 It is a flowchart of the method of the present invention;

[0117] Figure 2 This is a structural diagram of the static decoupling module in the present invention. DETAILED DESCRIPTION

[0118] In order to make the purpose, technical solutions and advantages of this application more clear, this application is further described in detail below with reference to the accompanying drawings and embodiments.

[0119] Example 1

[0120] The generative zero-shot learning method based on dual-state collaborative decoupling and semantic refinement described in this embodiment has a flow chart as follows: Figure 1 As shown, the method includes the following steps:

[0121] Step 1: Input the target modal data into the feature extraction module to extract the global modal features; input the global modal features into the static decoupling module to perform multi-level static decoupling and separation on the features to obtain static features of different levels.

[0122] Extract global visual features after image or audio modality conversion through pre-trained Visual Transformer (VIT) , the original visual features are preprocessed into , which will then retain the key information Input feature decomposition encoder, static decomposition into background level features , structural level features and level-of-detail features ; Finally, after cross-level interaction and feature optimization, the above features are output to the static decoupling module and the dynamic decoupling module.

[0123] like Figure 2 The structure diagram of the static decoupling module is shown. Further, step 1 includes:

[0124] The target modality data is visual modality or audio modality;

[0125] For the visual modality, the global visual feature is recorded as , the global visual features Convert to image label sequence through VIT , and add learnable category labels , forming a complete input feature ;

[0126] For the audio modality, the audio is converted into a Mel spectrum, and the time-frequency features of the Mel spectrum are enhanced through the fine-tuning module. The spectrum is then input into the VIT network to convert the audio modality into a visual modality. In this example, the fine-tuning model uses the Conformer neural network.

[0127] Compute input features through query, key, and value mapping functions The attention weight , the formula is:

[0128] ,

[0129] Where, represents the bond matrix, represents transpose, is the scaling factor, a is the head number; is the query vector of the i-th position, and the expression is , is the input feature matrix The i-th row of , N is the number of positions where the high-dimensional Mel spectrum or image is converted into one dimension, and d is the feature dimension. represents a learnable mapping function;

[0130] The value vector V of all spatial positions is calculated according to the corresponding attention weight Perform linear weighting and then aggregate along the spatial dimension to generate tuned features. The tuned features retain key semantic information and filter redundant noise. The formula is:

[0131] ,

[0132] Where, Represents element-wise multiplication;

[0133] Then upsample to restore the spatial dimension and get the feature , Indicates height, W indicates width, and C indicates the number of channels;

[0134] Through multi-level feature decomposition encoding, the features Decomposed into three levels of static features: background level features, structure level features and detail level features;

[0135] Among them, the low-frequency background information of the image is extracted by 1×1 convolution dimensionality reduction and 3×3 depth-separable convolution to extract the low-frequency background, and channel attention is introduced to enhance the key channel response. The expression of the background level feature is:

[0136] ,

[0137] Where, represents channel attention, represents a 3×3 depthwise separable convolution, Represents 1×1 convolution;

[0138] The dilated convolution is used to expand the receptive field to capture the contour, and the spatial attention (CBAM) is combined to focus on the geometric structure to capture the mid-frequency contour and geometric structure. The expression of the structural level feature is:

[0139] ,

[0140] Where, represents spatial attention, Represents 3×3 dilated convolution;

[0141] The residual block is used to enhance high-frequency textures, and the local self-attention is used to model long-range dependencies to preserve color and details. The expression of the detail-level feature is:

[0142] ,

[0143] Where, represents local self-attention, represents the residual block;

[0144] In order to avoid semantic discontinuity caused by complete separation after feature decoupling, the features of each level are cross-level interacted and optimized, and the background-level features and structure-level features are fused through cross-attention to eliminate background semantic ambiguity and update the background-level features. The formula is:

[0145] ,

[0146] Where, is the updated background-level feature, represents the cross attention mechanism;

[0147] The detail-level features are dynamically combined with the structure-level features through gated weighting to update the structure-level features. The formula is:

[0148] ,

[0149] Where, represents the updated structural features; Represents the gating coefficient, dynamically adjusts the contribution ratio of details and structures, and strengthens key areas (such as edge textures). The calculation formula is:

[0150] ,

[0151] Where, Represents the Sigmoid activation function.

[0152] In step 2, the global modal features and static features at different levels are input into the dynamic decoupling module. Through the channel attention mechanism, the shared dynamic patterns in the modal features are screened and extracted.

[0153] The dynamic decoupling module first extracts the global features of the image through the VIT and static decoupling modules. and static features , and use the spatial-channel attention mechanism to extract common patterns in static features and , further generate stable common prototypes through common prototype aggregation These prototypes serve as anchors to align common features within the same category while ensuring that individual features between categories are pushed further apart, enhancing inter-class differentiation. The final output is the refined dynamic common features as the pseudo-label representation of the image. The static features (background, structure, and details) decoupled in step 1 are then combined with the dynamic common features (such as posture and material) to achieve dual-state decoupling of static features (individuality) and dynamic features (commonality).

[0154] Furthermore, step 2 includes:

[0155] Global visual features Combined with multi-level static features into joint features , the formula is:

[0156] ,

[0157] Filtering joint features through spatial-channel attention High-frequency activation areas generate common features , the formula is:

[0158] ,

[0159] ,

[0160] Where, express activation function, is the attention mask, Represents a 3×3 convolution operation that focuses on common areas across categories, such as recurring poses or textures; Represents element-wise multiplication;

[0161] The common characteristics Decomposed into two types of dynamic visual features as dynamic patterns: structural dynamic features and detail dynamic features;

[0162] The receptive field is expanded by using a dilation factor of 22 to capture the common geometric structure across categories and obtain structural dynamic features such as posture and morphology. The expression is:

[0163] ,

[0164] By combining the residual block with non-local attention, long-range dependencies are modeled to extract high-frequency common textures and obtain detailed dynamic features such as material and light and shadow. The expression is:

[0165] ,

[0166] The output structural dynamic features and detail dynamic features are used as pseudo-label representations of the image.

[0167] Furthermore, in step 2, when decomposing the common features into two types of dynamic visual features, a new triplet loss function is constructed. The process includes:

[0168] Perform cross-sample mean pooling on dynamic features to generate common prototypes. The formula is:

[0169] ,

[0170] Where, represents the dynamic feature vector corresponding to the i-th sample, and N represents the total number of dynamic feature vectors participating in mean pooling;

[0171] Common prototype As an anchor , taking the personality characteristics of different instances of the same category as positive samples , among which personality traits Use static features directly or , personality traits of other categories are used as negative samples ;

[0172] Construct the loss function, the expression is:

[0173] , where m represents the interval hyperparameter in the loss, which is used to control the minimum separation between positive and negative samples and anchor points.

[0174] In order to further clarify the decoupling of commonality and individuality, different from the traditional triple loss, the new triple loss proposed in this example ensures By only representing the shared pattern itself, the unique features of the two categories are far away from the anchor point, which means that they are naturally distanced from each other, thereby increasing the distinction between categories and providing a way to construct a dynamic semantic space from low-level static features.

[0175] In step 3, static features and dynamic patterns at different levels are input into the prototype memory module to generate reusable feature combinations and provide initial feature prototypes for unseen classes. During the training process, the feature weights are dynamically adjusted based on the feedback from the confusion detection module, and a conditional vector fitting Gaussian noise is generated and input into the generative model.

[0176] Prototype memory module input static features With dynamic features , and aggregate dynamic features During training, the generated dynamic prototypes are updated to further align the features of the generated images with the real images. After training, the static features and their weights are combined to approximate traditional Gaussian noise using KL divergence. A portion of this noise is mixed and fed into the generative model as guidance. The weighted static and dynamic features are then output to the semantic refinement module to guide semantic updates.

[0177] Furthermore, step 3 includes:

[0178] The structural level features, detail level features, and background level features are directly used as the personality prototypes within the class, without participating in cross-class aggregation, and a static weight matrix is ​​generated through offline pre-training. , fixed static weight matrix after training , the formula is:

[0179] ,

[0180] Where, represents a multilayer perceptron with two linear layers, Represents the static prototype vector of the cth class, the formula is:

[0181] ,

[0182] Where, represents the number of samples in the cth class, represents the structural static features of the i-th sample of the c-th category, represents the detail-level static features of the i-th sample of the c-th category;

[0183] Each category is learned independently and fixed after training. , avoid online updates;

[0184] Perform cross-sample mean pooling on the structural dynamic features and detail dynamic features to generate dynamic commonality prototypes. The formula is:

[0185] ,

[0186] Where, represents the structural dynamic characteristics of the i-th sample of the c-th category, Represents the detailed dynamic features of the i-th sample of the c-th category;

[0187] Update on the prototype of generated data: dynamic features output by the generative model (CGAN) With static features Update the prototypes respectively, the formulas are:

[0188] ,

[0189] ,

[0190] Where, represents the intra-class mean of the dynamic features output by the generated model at the t-th iteration, Represents the intra-class mean of the static features output by the generated model at the t-th iteration;

[0191] The prototype alignment is then performed through a dynamic update mechanism, where Force the dynamic characteristics of the generated model to approach the real dynamic prototype, by Perform static feature alignment; where Represents a static prototype vector;

[0192] At the completion of training, the static features are noise-fitted, that is, the static prototypes and feature weights are combined into learnable distribution parameters, including the mean and variance , the formulas are:

[0193] ,

[0194] ,

[0195] Where, It means that it is calculated through a linear layer. represents the static characteristics of the cth class, , the joint posterior distribution is explicitly defined in terms of mean and variance, as follows:

[0196] ,

[0197] Introducing KL divergence constraint loss, the formula is:

[0198] ,

[0199] Where tr is the trace operation of the matrix, d is the dimension of the latent variable, and log det is the logarithm of the determinant;

[0200] Mix the original Gaussian noise to generate the conditional vector , the formula is:

[0201] ,

[0202] Where, is the weight, which controls the proportion of original noise retained. and are all represented as Gaussian noise.

[0203] In step 4, the dynamic pattern is input into the semantic refinement module, the corresponding semantic features are generated through the mirrored cross-attention mechanism, and the global modal features and semantic features are bidirectionally aligned to obtain the refined semantic representation.

[0204] To further improve the accuracy of semantic updates, a semantic feedback module is introduced. This module comprehensively evaluates the discriminator confidence, classifier entropy, and feature matching error, dynamically adjusts the weight distribution in the loss function, and achieves adaptive selection of semantic update timing.

[0205] The semantic refinement module combines the dynamic features output by the static decoupling module with the weighted static features and the original semantic features to form a joint visual feature. , and convert it into semantic features through a fixed mapping network , and then converted back into visual features through a fixed generator G Constraining and refining semantic features through mirror-symmetric visual-semantic cross-attention and semantic-visual cross-attention mechanisms , and finally output it to the generative model as a generation guide.

[0206] To overcome the limitations of fixed semantics in traditional zero-shot learning, the features in the adaptive prototype memory network are used to enrich the semantics. Furthermore, step 4 includes:

[0207] The weighted static features and dynamic features in the static decoupling module are spliced ​​into joint visual features. The formula is:

[0208] ,

[0209] The joint visual features are converted into initial semantic features through a fixed parameter mapping network R ;

[0210] The joint visual features are used as the query value in the attention mechanism, the initial semantic features are used as the key value and the true value, and the multi-granularity information in the visual features is captured through multi-head cross attention to obtain the enhanced semantic features. The formula is:

[0211] ,

[0212] Where, represents the query vector, represents the key vector, represents a value vector, represents the scaling factor;

[0213] Enhanced semantic features Then regenerate the visual features through the fixed parameter generation model (CGAN) ;

[0214] Semantic-visual cross attention is introduced, and the enhanced semantic features are used as query values. The newly generated visual features are used as key values ​​and true values. Multi-head cross attention is used to strengthen the multi-granularity connections in the semantic features, and the enhanced visual features are obtained. The formula is:

[0215] ,

[0216] Where, 、 Denote the query vector, key vector, and value vector, respectively. These two cross-attention modules form a mirror-symmetric structure, ensuring accurate and balanced bidirectional translation from vision to semantics and vice versa.

[0217] By cycle consistency constraints Ensure that the generated visual features are aligned with the original visual features, Represents the final visual feature vector and uses the loss function Further refine the semantic expression, Represents the cross attention output from visual to semantic direction, with original visual features Query key / value, with initial semantic features is the key / value, and the semantic representation obtained by attention calculation is recorded as , Represents the cross attention output from semantic to visual direction. The output of the dynamic semantic refinement network, that is, the refined semantic features, is consistent with the original semantic features in the dataset. They are used together in the subsequent generation process, thereby significantly improving the diversity and generalization ability of the generated features, overcoming the defects of traditional methods in zero-shot learning tasks such as fixed semantics and insufficient generalization.

[0218] In step 5, the refined semantic representation and conditional vector are used to synthesize samples of seen and unseen classes through the generative model, and then input into the zero-shot classifier to complete the comprehensive reasoning. During the reasoning process, based on the feedback of the classifier or discriminator, the same-class and cross-class confusion are identified, and the features in the prototype memory are updated through the confusion detection module. Finally, the updated semantic representation is passed into the generative model.

[0219] During the training process, the classifier predicts confidence and entropy values, and the confusion detection and alignment module judges classification errors (confusion). When the same type of confusion is detected, the detail weight is increased; when cross-class confusion is detected, the structure weight is increased.

[0220] Furthermore, step 5 includes:

[0221] First, the classifier ( Figure 1C) The confidence and entropy values ​​of the output are analyzed in real time. When the entropy value of the prediction result is greater than the threshold and fluctuates repeatedly between similar instances, it is judged as similar confusion. The loss makes the dynamic features of similar samples similar, and the static features and dynamic features of similar samples complementary; The loss formula is:

[0222] ,

[0223] Where, , The system automatically increases the focus on individual-level features, especially emphasizing the differences in detailed features. By increasing the individual-level feature contrast loss, the model's ability to distinguish subtle individual differences is enhanced.

[0224] When the classification result mistakenly falls into a category with significantly different semantics, it is judged as cross-class confusion. The loss increases the dynamic feature distance between samples of different classes, and the formula is:

[0225] ,

[0226] Where, is a predefined cross-class confusion pair, The interval hyperparameter automatically increases the weight of category-level features, such as structural or background features, and promotes the model's ability to distinguish categories at a more macro level through the category-level feature interval loss.

[0227] Utilize dynamic gating mechanism Adaptively generate attention weights for each feature level, and adjust the contribution of background features, structural features, and detail features in confusion recognition in real time. The formula is:

[0228] ,

[0229] Here, f uniformly represents the feature tensors extracted at different semantic levels, including background features, structural features, or detail features. These features are subsequently weighted through a dynamic gating mechanism, enabling the model to adaptively emphasize or suppress the contributions of the three types of features based on task requirements. The contribution of each feature in confusion recognition is adjusted through backpropagation, and the dynamic weight values ​​are fed back to the static decoupling module to further optimize the generation process.

[0230] Step 6: Combine the discriminator, classifier and feature matching error indicators, and use the semantic feedback module to dynamically evaluate the quality of the generated samples; based on the evaluation results, optimize the dynamic semantic refinement module and adjust the weights of each feature level.

[0231] Furthermore, in step 6, the dynamic evaluation network uses a multi-layer perceptron (MLP) with the discriminator confidence, classifier entropy and feature matching error as input, and outputs a dynamic threshold and loss weight adjustment coefficient: ,in, is the penalty coefficient for the dynamic threshold growth of the hyperparameter, used to suppress The threshold value is increased excessively between two adjacent iterations to ensure smooth evolution. is the dynamic threshold at the tth iteration; the dynamic threshold is used to determine whether to trigger the semantic feature update. When the discriminator confidence is less than the confidence threshold, the classifier entropy is greater than the entropy threshold, or the feature matching error exceeds the error threshold, the module automatically starts the optimization adjustment of the semantic feature. This module analyzes the prediction confidence of the classifier. and entropy , identify and distinguish between two typical situations: "same-class confusion" and "cross-class confusion", Represents the confidence of the i-th sample;

[0232] After triggering the semantic update, the module optimizes the cross-attention module parameters in the semantic refinement module through gradient backpropagation; at the same time, the loss weight adjustment coefficient is used to adaptively adjust the weight distribution of each loss item in the semantic refinement module. , thereby dynamically optimizing the semantic refinement process, where is the alignment loss weight coefficient, which controls the contribution of the visual-semantic alignment term in the weight redistribution formula. is the alignment score, which is obtained by calculating the cosine similarity between the refined semantic vector and the original semantic vector. The range is [0, 1]. The larger the value, the more consistent the two are. Represents the mirror consistency weight coefficient, which measures the impact of mirror cross attention (symmetry constraint) on the overall weight, Represents the entropy value of the classifier, the entropy of the predicted probability p obtained by the generated / real feature through the classifier , reflecting category uncertainty, Represents the feature distribution consistency weight coefficient, which adjusts the proportion of feature matching loss in the total weight. Represents the feature matching loss, which is the difference between the generated features and the real features in the feature space Distance is used to ensure the authenticity and distribution consistency of synthetic features.

[0233] Step 7: Repeat steps 3 to 6 and calculate the Top-1 until the iteration termination condition is met, and then stop the iteration to obtain the final neural network architecture, which includes a modal feature collaborative decoupling network and a dynamic semantic adaptive network.

[0234] Repeat steps 3 to 6 and calculate the top-1 classification accuracy of the zero-shot learning task on the validation set after each iteration. . When the accuracy gain is met for K consecutive iterations When both are less than a given threshold ε, the loop is terminated and the final model is output. K and ε can be set based on validation set experience or prior knowledge.

[0235] Step 8: During the testing phase, the synthesized unseen class samples are used to enhance the generalization ability of the classifier, ultimately achieving the goal of zero-shot learning and accurately identifying the modal data of seen and unseen classes.

[0236] To address the challenges of traditional generative zero-shot learning models, such as insufficient alignment between modal data (such as images and audio) and semantics, weak generalization, and low-quality generated samples, this embodiment proposes a generative zero-shot learning method based on dual-modal collaborative decoupling and semantic refinement. This approach constructs a neural network architecture combining a modal feature collaborative decoupling network and a dynamic semantic adaptive network. The modal feature collaborative decoupling network effectively achieves structured decoupling of modal features through multi-level feature decomposition and cross-modal common feature extraction, significantly reducing redundant information interference and enhancing cross-category generalization performance. The dynamic semantic adaptive network utilizes a mirrored cross-attention mechanism to achieve bidirectional dynamic alignment of modal and semantic features. It also dynamically regulates feature attention through adaptive confusion detection and semantic feedback mechanisms, effectively improving the quality and stability of generated samples. This overall architecture, centered on "modal feature collaborative decoupling + dynamic semantic adaptive regulation," overcomes the generalization bottleneck of existing generative zero-shot learning models and possesses strong theoretical value and broad application prospects.

[0237] To verify the effectiveness of the generative zero-shot learning method based on dual-state collaborative decoupling and semantic refinement proposed in this invention, this embodiment was tested on three standard datasets: AWA2, SUN, and CUB. Based on the test results, the Top-1 accuracy of traditional zero-shot learning methods and generalized zero-shot learning methods was evaluated. The AWA2 dataset contains 37,322 animal images, covering 50 categories, each of which is described by 85-dimensional attributes. According to the standard division, 40 classes are seen classes and 10 classes are unseen classes. The SUN dataset contains 14,340 scene images, covering 717 categories, each of which is described by 102-dimensional attributes. In the division, 645 classes are seen classes and 72 classes are unseen classes. The CUB dataset contains 11,788 scene images, covering 200 categories, each of which is described by 1024-dimensional attributes. In the division, 150 classes are seen classes and 50 classes are unseen classes. Traditional zero-shot learning (ZSL): Only the top-1 classification accuracy of unseen examples is tested. Generalized zero-shot learning (GZSL): Both seen and unseen examples are tested, and the accuracy of the seen class (S) and the accuracy of the unseen class (U) are calculated, as well as the harmonic mean H = (2×S×U) / (S+U).

[0238] To further verify the generalizability of the proposed method across audio modalities, the experiment was expanded to the ESC-50 environmental audio dataset. ESC-50 contains 2,000 5-second environmental audio clips covering 50 acoustic categories, such as "dog barking," "thunderstorm," and "mechanical noise," with 40 clips per category. Following the conventions of zero-shot learning, a 50-fold split was used: 40 categories were randomly selected as seen categories, and the remaining 10 categories were unseen categories. The training, validation, and testing phases were guaranteed to have no overlapping categories.

[0239] As shown in Table 1 (unit: %), various generative and non-generative zero-shot learning methods are listed. Under the traditional zero-shot learning setting, the proposed method achieves top-1 classification accuracy of 83.2%, 77.3%, and 86.5% on the AWA2, SUN, and CUB datasets, respectively. Under the generalized zero-shot learning setting, the top-1 classification accuracy for the AWA2 dataset is 81.7% for seen classes, 75.6% for unseen classes, and 78.5% for the harmonic mean. For the SUN dataset, the top-1 classification accuracy is 62.4% for seen classes, 47.6% for unseen classes, and 53.9% for the harmonic mean. For the CUB dataset, the top-1 classification accuracy is 77.8% for seen classes, 73.6% for unseen classes, and 75.6% for the harmonic mean. On the ESC-50 ambient audio dataset, where traditional zero-shot learning is evaluated on only 10 unseen classes, the proposed method achieves a 30.9% accuracy. In the generalized zero-shot learning test of all 50 categories, the proposed method achieved an accuracy of 55.8% for seen categories and 24.6% for unseen categories, with a harmonic mean H = 34.1%.

[0240] Table 1

[0241]

[0242] Example 2

[0243] The generative zero-shot learning system based on dual-state collaborative decoupling and semantic refinement described in this embodiment includes:

[0244] The static decoupling module tunes the global mode through the preprocessing module; the feature decomposition encoder further performs three-level static decoupling of the tuned features at the background level, structure level, and detail level;

[0245] Dynamic decoupling module, used to further extract common patterns from global modal features and static features to achieve dual-state collaborative decoupling of static and dynamic features;

[0246] The prototype memory module is used to dynamically update static and dynamic features. By dynamically adjusting feature weights, it generates a conditional vector that fits Gaussian noise and provides guidance for feature generation.

[0247] The semantic refinement module uses mirror-symmetric cross-attention to achieve bidirectional refinement and update of visual and semantic features to obtain refined semantic representations;

[0248] The confusion detection module analyzes the same-class and cross-class confusion based on the feedback of the classifier prediction, and generates and adjusts the attention weights of each feature level in real time through a dynamic gating mechanism;

[0249] The semantic feedback module automatically triggers the semantic update timing of the semantic refinement module based on the generation quality by comprehensively evaluating the discriminator confidence, classifier entropy and feature matching error;

[0250] The system further includes a loss function optimization strategy, including alignment loss, mirror symmetry loss, distribution consistency loss and feature matching loss, and dynamically adjusts the weights of each loss term according to the semantic feedback module, optimizes the parameters of each module through backpropagation, and iterates repeatedly until the generation quality reaches a preset threshold; ultimately, high-quality generated visual features are output, which significantly enhances the generalization ability of the zero-shot learning classifier and achieves high-precision reasoning and recognition of seen and unseen classes, overcoming the generalization bottleneck and domain shift problems of static semantics in traditional zero-shot learning methods, significantly improving the diversity and generalization ability of generated features, and achieving an effective breakthrough in zero-shot learning performance through the two-way collaborative update of visual features and semantic features.

Claims

1. A generative zero-shot learning method based on dual-state collaborative decoupling and semantic refinement, characterized by: The steps include: Step 1: Input the target modal data into the feature extraction module to extract the global modal features; input the global modal features into the static decoupling module to perform multi-level static decoupling and separation on the features to obtain static features of different levels; Step 2: The global modal features and static features at different levels are input into the dynamic decoupling module. The channel attention mechanism is used to filter and extract the shared dynamic patterns in the modal features. Step 3: Input static features and dynamic patterns at different levels into the prototype memory module to generate reusable feature combinations and provide initial feature prototypes for unseen classes. During the training process, the feature weights are dynamically adjusted based on the feedback from the obfuscation detection module, and a conditional vector fitting Gaussian noise is generated and input into the generative model. Step 4: Input the dynamic pattern into the semantic refinement module, generate corresponding semantic features through the mirrored cross-attention mechanism, and align the global modal features with the semantic features to obtain the refined semantic representation; Step 5: Using the refined semantic representation and conditional vector, the generative model synthesizes samples of seen and unseen classes, and inputs them into the zero-shot classifier to complete comprehensive reasoning. During the reasoning process, based on the feedback from the classifier or discriminator, the same-class and cross-class confusion are identified, and the features in the prototype memory are updated through the confusion detection module. Finally, the updated semantic representation is input into the generative model. Step 6: Combine the discriminator, classifier, and feature matching error indicators and use the semantic feedback module to dynamically evaluate the quality of the generated samples. Based on the evaluation results, the dynamic semantic refinement module is optimized to adjust the weights of each feature level. Step 7: Repeat steps 3 to 6 and calculate the top-1 until the iteration termination condition is met to obtain the final neural network architecture. Step 8: During the testing phase, the synthesized unseen class samples are used to enhance the generalization ability of the classifier, ultimately achieving the goal of zero-shot learning and accurately identifying the modal data of seen and unseen classes.

2. The generative zero-shot learning method based on dual-state collaborative decoupling and semantic refinement according to claim 1 is characterized in that: Step 1 includes: The target modality data is visual modality or audio modality; for the visual modality, the global visual features are recorded as , the global visual features Converting to image-labeled sequences via a visual transformer , and add learnable category labels , forming a complete input feature ; For the audio modality, the audio is converted into a Mel spectrum, the time-frequency features of the Mel spectrum are enhanced through the fine-tuning module, and the result is input into the visual Transformer to convert the audio modality into the visual modality. Compute input features through query, key, and value mapping functions The attention weight , the formula is: , where is the query vector of the i-th position, and the expression is , is the input feature matrix The i-th row of N is the number of positions, d is the feature dimension, represents a learnable mapping function; represents the bond matrix, represents transpose, is the scaling factor, a is the head number; The value vector V of all spatial positions is calculated according to the corresponding attention weight Perform linear weighting and then aggregate along the spatial dimension to generate tuned features , Represents element-by-element multiplication; and upsampling restores the spatial dimension to obtain features , Indicates height, W indicates width, and C indicates the number of channels; Through multi-level feature decomposition encoding, the features Decomposed into three levels of static features: background level features, structure level features and detail level features; Among them, the expression of background level features is: ， Where, represents channel attention, represents a 3×3 depthwise separable convolution, Represents 1×1 convolution; The expression of the structural level feature is: ， Where, represents spatial attention, Represents 3×3 dilated convolution; The expression of detail level feature is: ， Where, represents local self-attention, represents the residual block; The background-level features and structure-level features are fused through cross attention to update the background-level features. The formula is: ， Where, is the updated background-level feature, represents the cross attention mechanism; The detail-level features are dynamically combined with the structure-level features through gated weighting to update the structure-level features. The formula is: ， Where, represents the updated structural features, Represents the gating coefficient, and the calculation formula is: ， Where, Represents the Sigmoid activation function.

3. The generative zero-shot learning method based on dual-state collaborative decoupling and semantic refinement according to claim 2, characterized in that: Step 2 includes: Global visual features Combined with multi-level static features into joint features , the formula is: ， Filtering joint features through spatial-channel attention The high-frequency activation area generates common features, and the formula is: ， ， Where, express activation function, is the attention mask, Represents common characteristics, Represents a 3×3 convolution operation; Decompose common features into two types of dynamic visual features as dynamic patterns: structural dynamic features and detail dynamic features; The expression of the structural dynamic characteristics is: ， The expression of the detailed dynamic characteristics is: ， The output structural dynamic features and detail dynamic features are used as pseudo-label representations of the image.

4. The generative zero-shot learning method based on dual-state collaborative decoupling and semantic refinement according to claim 3 is characterized in that: In step 2, when decomposing the common features into two types of dynamic visual features, a new triplet loss function is constructed. The process includes: Perform cross-sample mean pooling on dynamic features to generate common prototypes. The formula is: ， Where, represents the dynamic feature vector corresponding to the i-th sample, and N represents the total number of dynamic feature vectors participating in mean pooling; Common prototype As an anchor , taking the personality characteristics of different instances of the same category as positive samples , among which personality traits Use static features directly or , personality traits of other categories are used as negative samples ; Construct the loss function, the expression is: , where m represents the interval hyperparameter in the loss, which is used to control the minimum separation between positive and negative samples and anchor points.

5. The generative zero-shot learning method based on dual-state collaborative decoupling and semantic refinement according to claim 4, characterized in that: Step 3 includes: The structural level features, detail level features, and background level features are directly used as the personality prototypes within the class, without participating in cross-class aggregation, and a static weight matrix is ​​generated through offline pre-training. , fixed static weight matrix after training , the formula is: ， Where, represents a multilayer perceptron with two linear layers, Represents the static prototype vector of the cth class, the formula is: ， Where, represents the number of samples in the cth class, represents the structural static features of the i-th sample of the c-th category, represents the detail-level static features of the i-th sample of the c-th category; Perform cross-sample mean pooling on the structural dynamic features and detail dynamic features to generate dynamic commonality prototypes. The formula is: ， Where, represents the structural dynamic characteristics of the i-th sample of the c-th category, Represents the detailed dynamic features of the i-th sample of the c-th category; Update the generated data prototype using the dynamic features output by the generated model With static features Update prototypes separately , , where represents the intra-class mean of the dynamic features output by the generated model at the t-th iteration, represents the intra-class mean of the static features output by the generated model at the tth iteration; The prototype alignment is then performed through a dynamic update mechanism, where Force the dynamic characteristics of the generated model to approach the real dynamic prototype, by Perform static feature alignment; where Represents a static prototype vector; At the completion of training, the static features are noise-fitted, that is, the static prototypes and feature weights are combined into learnable distribution parameters, including the mean and variance , the formulas are: ， ， Where, It means that it is calculated through a linear layer. represents the static characteristics of the cth class, , the joint posterior distribution is explicitly defined in terms of mean and variance, as follows: ， Introducing KL divergence constraint loss, the formula is: ， Where tr is the trace operation of the matrix, d is the dimension of the latent variable, and log det is the logarithm of the determinant; Mix the original Gaussian noise to generate the conditional vector , the formula is: ， Where, is the weight, which controls the proportion of original noise retained. and are all represented as Gaussian noise.

6. The generative zero-shot learning method based on dual-state collaborative decoupling and semantic refinement according to claim 5, characterized in that: Step 4 includes: The weighted static features and dynamic features in the static decoupling module are spliced ​​into joint visual features. The formula is: ， The joint visual features are converted into initial semantic features through a fixed parameter mapping network R ; The joint visual features are used as the query value in the attention mechanism, the initial semantic features are used as the key value and the true value, and the multi-granularity information in the visual features is captured through multi-head cross attention to obtain the enhanced semantic features. The formula is: ， Where, represents the query vector, represents the key vector, represents a value vector, represents the scaling factor; The enhanced semantic features are then used to regenerate visual features through a fixed parameter generation model ; Semantic-visual cross attention is introduced, and the enhanced semantic features are used as query values. The newly generated visual features are used as key values ​​and true values. Multi-head cross attention is used to strengthen the multi-granularity connections in the semantic features, and the enhanced visual features are obtained. The formula is: ， Where, 、 Represent the query vector, key vector and value vector respectively.

7. The generative zero-shot learning method based on dual-state collaborative decoupling and semantic refinement according to claim 6, characterized in that: Step 5 includes: First, the confidence and entropy values ​​of the classifier output are analyzed in real time. When the entropy value of the prediction result is greater than the threshold and fluctuates repeatedly between similar instances, it is judged as similar confusion. The loss makes the dynamic features of similar samples similar, and the static features and dynamic features of similar samples complementary; The loss formula is: ， Where, , ; Among them, when the classification result mistakenly falls into a category with significantly different semantics, it is judged as cross-class confusion, and the The loss increases the dynamic feature distance between samples of different classes, and the formula is: ， Where, is a predefined cross-class confusion pair, is the interval hyperparameter; Utilize dynamic gating mechanism Adaptively generate the attention weights of each feature level, the formula is: , where f uniformly represents the feature tensor extracted at different semantic levels.

8. The generative zero-shot learning method based on dual-state collaborative decoupling and semantic refinement according to claim 7, characterized in that: In step 6, the dynamic evaluation network uses a multi-layer perceptron, takes the discriminator confidence, classifier entropy and feature matching error as input, and outputs a dynamic threshold and loss weight adjustment coefficient: ,in, is the penalty coefficient for the dynamic threshold growth of the hyperparameter, is the dynamic threshold at the tth iteration; the dynamic threshold is used to determine whether to trigger the semantic feature update. When the discriminator confidence is less than the confidence threshold, the classifier entropy is greater than the entropy threshold, or the feature matching error exceeds the error threshold, the module automatically starts the optimization adjustment of the semantic features. After triggering the semantic update, the parameters of the cross-attention module in the semantic refinement module are optimized through gradient backpropagation. At the same time, the weight distribution of each loss item in the semantic refinement module is adaptively adjusted using the loss weight adjustment coefficient. The formula is: , where is the alignment loss weight coefficient, is the alignment score, which ranges from [0, 1]. The larger the value, the more consistent the two are. represents the mirror consistency weight coefficient, represents the classifier entropy value, represents the feature distribution consistency weight coefficient, represents the feature matching loss.

9. A generative zero-shot learning system based on dual-state collaborative decoupling and semantic refinement, characterized by: The system comprises: The static decoupling module tunes the global mode through preprocessing operations; the feature decomposition encoder further performs three-level static decoupling of the tuned features at the background level, structure level, and detail level; Dynamic decoupling module, used to further extract common patterns from global modal features and static features to achieve dual-state collaborative decoupling of static and dynamic features; The prototype memory module is used to dynamically update static and dynamic features. By dynamically adjusting feature weights, it generates a conditional vector that fits Gaussian noise and provides guidance for feature generation. The semantic refinement module uses mirror-symmetric cross-attention to achieve bidirectional refinement and update of visual and semantic features to obtain refined semantic representations; The confusion detection module analyzes the same-class and cross-class confusion based on the feedback of the classifier prediction, and generates and adjusts the attention weights of each feature level in real time through a dynamic gating mechanism; The semantic feedback module automatically triggers the semantic update timing of the semantic refinement module based on the generation quality by comprehensively evaluating the discriminator confidence, classifier entropy and feature matching error.

10. The generative zero-shot learning system based on dual-state collaborative decoupling and semantic refinement according to claim 9, characterized in that: The system includes alignment loss, mirror symmetry loss, distribution consistency loss and feature matching loss, and dynamically adjusts the weights of each loss item according to the semantic feedback module, optimizes the parameters of each module through back propagation, and iterates until the generation quality reaches a preset threshold; finally, it outputs high-quality generated visual features.

Images