A zero-shot image description generation method based on spectral domain purification degree alignment
By removing visual noise through spectral domain purification and cascaded degree-of-freedom alignment mechanisms, the problem of visual noise interference in zero-sample image description is solved, achieving high-precision and robust image description generation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- FUZHOU UNIV
- Filing Date
- 2026-06-23
- Publication Date
- 2026-07-24
AI Technical Summary
Existing zero-sample image description techniques cannot effectively separate visual noise and achieve multi-dimensional depth alignment, resulting in grammatical errors and semantic deviations in the generated text.
By employing spectral domain purification and cascaded degree-of-freedom alignment mechanisms, visual-specific interference is removed and multi-dimensional semantic alignment is achieved. This includes pure text corpus processing during the training phase and visual feature spectral domain purification, dynamic signal-to-noise ratio calibration, multi-level cascaded mapping, and projection operations during the inference phase.
Without requiring paired data, it significantly improves the generation accuracy and cross-domain robustness of image descriptions, ensuring the semantic consistency and accuracy of the generated text.
Smart Images
Figure CN122454576A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the interdisciplinary field of computer vision and natural language processing, and specifically relates to a zero-sample image description generation method based on spectral domain purification degree of freedom alignment in multimodal generation tasks. Background Technology
[0002] Zero-shot image captioning techniques aim to generate natural language descriptions for input images without requiring any paired image-text training data. Existing methods typically utilize pre-trained visual language models (such as CLIP) to extract features and feed them into large language models (LLMs) trained on plain text.
[0003] However, pre-trained visual features are inherently complex signals, inevitably coupled with a large amount of high-variance, modality-specific noise (such as texture, lighting, and viewpoint) unrelated to text generation. Since language models are optimized only on pure text corpora, this visual noise leads to severe semantic interference. Existing methods generally employ shallow global mappings, which cannot separate these orthogonal noises nor provide sufficient degrees of freedom for representing heterogeneous features. In deep space, these insufficiently exposed and corrected features quickly deviate from the semantic manifold of the text, resulting in grammatical errors and semantic deviations in the generated text. Currently, there is a lack of an architecture in the field that can explicitly purify noise and achieve multi-dimensional depth alignment. Summary of the Invention
[0004] The purpose of this invention is to propose a zero-sample image description generation method based on spectral domain purification degree of freedom alignment, so as to overcome the shortcomings of existing global mapping paradigms that cannot separate complex visual signals and have insufficient representation degrees of freedom. It can remove visual-specific interference and achieve multi-dimensional semantic alignment without the need for image-text pairing data.
[0005] To achieve the above objectives, the technical solution of this invention is: a zero-sample image description generation method based on spectral domain purified degrees of freedom alignment, comprising a training phase and an inference phase:
[0006] The training phase is based on training large-scale language models and cascaded projection networks using plain text corpora.
[0007] The reasoning phase includes the following steps:
[0008] Obtain the raw visual features of the target image;
[0009] The original visual features are purified by spectral domain purification based on modal covariance comparison and generalized Fourier transform to reconstruct semantic reference features;
[0010] Spatial residual feature fusion based on dynamic signal-to-noise ratio calibration is performed on semantic reference features and original visual features to generate purified output features;
[0011] The purified output features are input into a multi-level cascaded degree-of-freedom compensation and alignment network, which outputs calibrated and purified features. The cascaded degree-of-freedom compensation and alignment network performs parameterized linear mapping and non-parameterized memory projection operations sequentially at each level. The linear layer weights used in the parameterized linear mapping operation are initialized by the linear layer weights fixed after training in the cascaded projection network. The parameter weights fixed during training provide representational degrees of freedom for the features, aiming to expose potential semantic features in multiple dimensions, thereby enabling subsequent calibration and alignment to be implemented across a wider range of feature dimensions. The cascaded degree-of-freedom compensation and alignment, through iterative multi-level cascaded linear layers and memory retrieval, dynamically calibrates while progressively increasing the feature exposure dimensions, thereby achieving comprehensive and consistent alignment between heterogeneous features and the text manifold within the deep semantic space.
[0012] The calibrated and purified features are input into a large language model for interference-free autoregressive decoding, and the target image description is output.
[0013] Preferably, the training of the large-scale language model and cascaded projection network based on plain text corpus is as follows:
[0014] Global text representations of plain text samples are extracted using a pre-trained and frozen text encoder;
[0015] The global text representation is sequentially passed through a cascaded projection network containing parameterized linear mapping and L2 normalization operations to generate prefix lexical units;
[0016] Prefix lexical units are concatenated to the front of the full lexical sequence of the plain text sample and input into a large language model (LLM). Autoregressive cross-entropy loss is used to jointly optimize the cascaded projection network and the large language model, thereby solidifying a plain text semantic manifold benchmark that is completely isolated from visual interference within the network.
[0017] Preferably, the calculation of the autoregressive cross-entropy loss is as follows:
[0018]
[0019] in, Represents the loss function. Indicates the first The target text terms that need to be predicted at each time step Indicates the first The sequence of historical text terms generated before a certain time step. This refers to the prefix words that are concatenated at the beginning of the sequence after being generated by the cascaded projection network. This represents the learnable model parameters in a cascaded projection network. This represents the learnable model parameters in a large language model. Indicates that in a given prefix word Historical text lexical sequence and model parameters and Under the condition that the network is in the first Step-by-step generation of target lexical units The conditional probability.
[0020] Preferably, the reconstruction of the semantic reference features is as follows:
[0021] Extracting raw visual features from target images using a pre-trained and frozen image encoder. ;
[0022] Using the pre-solved optimal orthogonal basis, the original visual features are processed. Perform a generalized Fourier transform and project it onto the spectral domain, based on a preset cumulative energy retention rate. After truncating the noise dimension corresponding to non-positive feature values, an inverse generalized Fourier transform is performed to reconstruct the initially purified semantic reference features. .
[0023] Preferably, the solution for the optimal orthogonal basis is as follows:
[0024] Using pre-trained and frozen text encoders and image encoders, text sample sets and visual sample sets are extracted from plain text corpora and image databases, respectively, where there is no one-to-one correspondence. The uncentered covariance matrix of the text sample set is calculated. The uncentered covariance matrix of the visual sample set ;
[0025] Combined with contrast scaling factor Constructing the contrastive Hamiltonian operator The optimal orthogonal basis is obtained by solving the comparative Hamiltonian operator.
[0026] Preferably, the generation of the purified output features is as follows:
[0027] The energy ratio is obtained by calculating the ratio of semantically preserved energy to truncated noise energy in the spectral domain. ;
[0028] Using the logarithmic mapping formula Calculate translation weights And use translation weights to adjust the original visual features With semantic reference features Perform residual weighted fusion and L2 normalization to generate purified output features. .
[0029] Preferably, in each level of the cascaded degree-of-freedom compensation alignment network:
[0030] Features processed by parameterized linear mapping operation as query vector Calculate the query vector and store it in the corresponding level of text support memory. The similarity between samples in the dataset is used to obtain the weight distribution after Softmax normalization;
[0031] Text support memory based on weight distribution The similarity of each sample is weighted and aggregated, and the features are then combined. Adjust towards the aggregation result, forcing anchoring to the target text manifold:
[0032]
[0033]
[0034] in, This indicates that the query vector and text support memory are used to support the query. The aggregated feature vector is obtained by weighted aggregation of the similarity of each sample. For temperature hyperparameters, This indicates the L2 normalization operation. Indicates the first Hierarchical output features This represents a fixed hyperparameter used to control the strength of residual connections, i.e., the residual factor.
[0035] Preferably, the interference-free autoregressive decoding is as follows:
[0036] The spectral domain purification process and cascaded degree of freedom compensation alignment mechanism are triggered and completed only during the pre-filling stage of model inference.
[0037] After calibration, the features are input into the large language model during the pre-filling stage. After forward computation by the large language model, key-value pair states of each attention layer are generated and cached.
[0038] In the subsequent autoregressive decoding stage, the large language model directly reuses the key-value pair state (KV-Cache) for word-by-word meta-generation until a complete target image description is output, without applying any cross-modal feature transformation calculations during this period.
[0039] Compared with the prior art, the present invention has the following beneficial effects:
[0040] This invention does not rely on paired data at all, decoupling visual noise from the bottom layer through spectral domain purification; by extending the degrees of freedom of cascaded multi-layer representations, it solves the problem of dimensionality limitation in feature alignment, achieving semantic calibration across more comprehensive dimensions. This framework significantly improves the generation accuracy and cross-domain robustness of image descriptions with extremely low overhead. Attached Figure Description
[0041] Figure 1 This is a flowchart of the method of the present invention;
[0042] Figure 2 This is a schematic diagram of the overall structure of the method of the present invention. Detailed Implementation
[0043] The following is in conjunction with the appendix Figure 1-2 The technical solution of the present invention will be described in detail below.
[0044] This invention provides a zero-sample image description generation method based on spectral domain purification and degree-of-freedom alignment. By constructing a spectral domain purification module and a cascaded degree-of-freedom compensation and alignment mechanism, visually specific noise is removed from the underlying layer, achieving multi-dimensional semantic alignment. Figure 1 As shown, this method includes the following steps:
[0045] 1) Training of a large-scale language model foundation and projection network based on plain text corpora;
[0046] 2) Inference stage: spectral domain purification based on modal covariance comparison and generalized Fourier transform;
[0047] 3) Spatial residual feature fusion based on dynamic signal-to-noise ratio calibration;
[0048] 4) Cascaded degree-of-freedom compensation alignment based on alternating execution of multi-level parametric mapping and non-parametric memory projection;
[0049] 5) Insulation-free autoregressive decoding output based on calibrated and purified features.
[0050] The following is a detailed implementation process of the present invention.
[0051] 1. Cross-modal foundation training process. In the training phase, which is completely independent of paired text-image data, this invention only observes a plain text corpus. Given a text sequence... First, a global text representation is extracted using a pre-trained and frozen CLIP text encoder. This representation is then sequentially passed through a cascaded projection network consisting of linear mapping layers and L2 normalization layers, aiming to align feature dimensions and initially construct the physical boundaries of the pure text semantic distribution. The processed text features are then used as visual prefixes, concatenated at the front of the original lexical sequence, and input into a large-scale language model. During training, the projection network and the language model are jointly optimized using cross-entropy loss modeled by standard autoregression.
[0052]
[0053] This process forces the model to solidify within a large language model, forming a robust pure text semantic manifold benchmark.
[0054] 2. During the reasoning stage, the original visual features of the input... regarded as core semantic components Modal-specific visual noise The system generates a composite signal. To remove these high-variance interferences unrelated to text generation from the underlying layer, this invention introduces a spectral domain purification mechanism. The system calculates the uncentered covariance matrix of the unpaired text distribution and the visual distribution. and And construct a contrastive Hamiltonian operator by combining contrastive scaling factors. Solving this operator yields the optimal orthogonal basis that maximizes net semantic energy. The input visual features are projected onto the spectral domain using a generalized Fourier transform (GFT). Within the spectral domain, the cumulative energy retention rate is calculated based on a preset value. The dimension representing visual-specific noise is truncated, and an inverse generalized Fourier transform (iGFT) is performed to obtain preliminarily purified semantic reference features in the reconstruction space. .
[0055] 3. Dynamic Spatial Feature Fusion Based on Energy Ratio. Since hard truncation in the spectral domain may damage some fragile semantic cues, this invention introduces a spatial residual interpolation mechanism. The endogenous energy ratio is defined by calculating the ratio of the semantic energy retained in the spectral domain to the truncated visual noise energy. The ratio was then converted into a shift weight using the logarithmic mapping formula. Using this weight to analyze the original visual features With reconstruction features Dynamic weighted fusion and L2 normalization are performed to generate purified features. This step ensures that key visual anchor information in the image is adaptively preserved while suppressing noise.
[0056] 4. A Multidimensional Compensation Alignment Mechanism Based on Cascaded Degrees of Freedom Expansion. To address the issue of limited feature dimensions in zero-sample alignment, this invention introduces a cascaded compensation mechanism, achieving depth calibration through multi-layered cascaded mapping and projection operations. In each layer, the system strictly alternates between two core operations: first, parametric linear mapping is performed. This operation provides the necessary representational degrees of freedom for the features using parameter weights fixed during training, aiming to expose the latent semantic feature components in the deep network in multiple dimensions. Immediately afterwards, non-parametric memory projection is performed. The system uses the features expanded with the aforementioned degrees of freedom as query vectors and retrieves them from the corresponding text support memory. By calculating Softmax similarity weights and weighted aggregation of samples in the support memory, the exposed multidimensional features are forcibly anchored to the target text manifold. This design, which provides degrees of freedom and exposes features through cascaded linear mappings and combines memory projection with "dynamic correction," allows alignment to be implemented across a wider and more comprehensive range of features.
[0057] 5. Efficient Output Generation Based on Calibrated Intermediate Activation Representations. All spectral purification and cascade compensation calculations are confined to the pre-filling stage of inference. In the subsequent autoregressive decoding stage, the large language model directly reuses the calibrated and mapped intermediate activation representations (i.e., key-value pair states) without applying any cross-modal feature transformation calculations. By decoupling visual interference and achieving precise multi-dimensional alignment, the model can output high-quality, semantically consistent image-based descriptive text end-to-end, with extremely low computational overhead throughout the generation process.
[0058] The above are preferred embodiments of the present invention. Any changes made to the technical solution of the present invention that do not exceed the scope of the technical solution of the present invention shall fall within the protection scope of the present invention.
Claims
1. A method for generating zero-sample image descriptions based on spectral domain purified degrees of freedom alignment, characterized in that, Includes a training phase and an inference phase: The training phase is based on training large-scale language models and cascaded projection networks using plain text corpora. The reasoning phase includes the following steps: Obtain the raw visual features of the target image; The original visual features are purified by spectral domain purification based on modal covariance comparison and generalized Fourier transform to reconstruct semantic reference features; Spatial residual feature fusion based on dynamic signal-to-noise ratio calibration is performed on semantic reference features and original visual features to generate purified output features; The purified output features are input into a multi-level cascaded degree-of-freedom compensation and alignment network, which outputs calibrated and purified features. The cascaded degree-of-freedom compensation and alignment network performs parameterized linear mapping and non-parameterized memory projection operations in each level. The linear layer weights used in the parameterized linear mapping operation are initialized by the linear layer weights that are fixed after training of the cascaded projection network during the training phase. The calibrated and purified features are input into a large language model for interference-free autoregressive decoding, and the target image description is output.
2. The zero-sample image description generation method based on spectral domain purified degrees of freedom alignment according to claim 1, characterized in that, The training of a large-scale language model and cascaded projection network based on plain text corpora is as follows: Global text representations of plain text samples are extracted using a pre-trained and frozen text encoder; The global text representation is sequentially passed through a cascaded projection network containing parameterized linear mapping and L2 normalization operations to generate prefix lexical units; Prefix lexical units are concatenated to the front of the full lexical sequence of the plain text sample and input into a large language model; autoregressive cross-entropy loss is used to jointly optimize the cascaded projection network and the large language model.
3. The zero-sample image description generation method based on spectral domain purified degrees of freedom alignment according to claim 2, characterized in that, The calculation of the autoregressive cross-entropy loss is as follows: in, Represents the loss function. Indicates the first The target text terms that need to be predicted at each time step Indicates the first The sequence of historical text terms generated before a certain time step. This refers to the prefix words that are concatenated at the beginning of the sequence after being generated by the cascaded projection network. This represents the learnable model parameters in a cascaded projection network. This represents the learnable model parameters in a large language model. Indicates that in a given prefix word Historical text lexical sequence and model parameters and Under the condition that the network is in the first Step-by-step generation of target lexical units The conditional probability.
4. The zero-sample image description generation method based on spectral domain purified degrees of freedom alignment according to claim 1, characterized in that, The reconstruction of the semantic reference features is as follows: Extracting raw visual features from target images using a pre-trained and frozen image encoder. ; Using the pre-solved optimal orthogonal basis, the original visual features are processed. Perform a generalized Fourier transform and project it onto the spectral domain, based on a preset cumulative energy retention rate. After truncating the noise dimension corresponding to non-positive feature values, an inverse generalized Fourier transform is performed to reconstruct the initially purified semantic reference features. .
5. The zero-sample image description generation method based on spectral domain purified degrees of freedom alignment according to claim 4, characterized in that, The solution for the optimal orthogonal basis is as follows: Using pre-trained and frozen text encoders and image encoders, text sample sets and visual sample sets are extracted from plain text corpora and image corpora where there is no one-to-one correspondence. Calculate the uncentered covariance matrix of the text sample set. The uncentered covariance matrix of the visual sample set ; Combined with contrast scaling factor Constructing the contrastive Hamiltonian operator The optimal orthogonal basis is obtained by solving the comparison Hamiltonian operator.
6. The zero-sample image description generation method based on spectral domain purified degrees of freedom alignment according to claim 4, characterized in that, The generation of the purified output features is as follows: The energy ratio is obtained by calculating the ratio of semantically preserved energy to truncated noise energy in the spectral domain. ; Using the logarithmic mapping formula Calculate translation weights And use translation weights to adjust the original visual features With semantic reference features Perform residual weighted fusion and L2 normalization to generate purified output features. .
7. The zero-sample image description generation method based on spectral domain purified degrees of freedom alignment according to claim 6, characterized in that, In each level of the cascaded degree-of-freedom compensation alignment network: Features processed by parameterized linear mapping operation As a query vector, the query vector is calculated and compared with the text support memory stored at the corresponding level. The similarity between samples in the dataset is used to obtain the weight distribution after Softmax normalization; Text support memory based on weight distribution The similarity of each sample is weighted and aggregated, and the features are then combined. Adjust towards the aggregation result, forcing anchoring to the target text manifold: in, This indicates that the query vector and text support memory are used to support the query. The aggregated feature vector is obtained by weighted aggregation of the similarity of each sample. For temperature hyperparameters, This indicates the L2 normalization operation. Indicates the first Hierarchical output features This represents a fixed hyperparameter used to control the strength of residual connections, i.e., the residual factor.
8. The method for generating zero-sample image descriptions based on spectral domain purified degrees of freedom alignment according to claim 2, characterized in that, The interference-free autoregressive decoding is as follows: The spectral domain purification process and cascaded degree of freedom compensation alignment mechanism are triggered and completed only during the pre-filling stage of model inference. After calibration, the features are input into the large language model during the pre-filling stage. After forward computation by the large language model, key-value pair states of each attention layer are generated and cached. In the subsequent autoregressive decoding stage, the large language model directly reuses the key-value pair state to generate word-by-word meta-generators until a complete description of the target image is output.