A zero-shot image description method based on synthetic image retrieval

Abstract

The application provides a zero-shot image description method based on synthetic image retrieval. A synthetic image set is generated by pre-training a text-to-image model, and samples with high consistency of image and text semantics are screened out. High consistency images are used as visual queries to retrieve relevant text features from a training corpus. The original text features, image features and retrieved text features are input into a feature fusion network, which models through an attention mechanism and outputs a conditional prefix representation. An autoregressive language model is used to decode and generate a description result and optimize model parameters. During inference, real images are used to replace synthetic images to perform the above retrieval and fusion process, obtain a target conditional prefix representation, and then generate a text description of the real image. The application effectively alleviates visual bias and improves description accuracy in a zero-shot scenario.

Description

Technical Field

[0001] This invention relates to an image recognition technology, and more particularly to a zero-sample image description method based on synthetic image retrieval. Background Technology

[0002] Image captioning aims to generate accurate and fluent text descriptions for images, representing a crucial research direction bridging computer vision and natural language processing. Traditional image captioning methods typically rely on large-scale image-text pairing datasets for supervised training. While achieving good results on standard datasets, their heavy dependence on manually labeled data limits their application in scenarios with limited annotation resources or significant variations in data distribution. Zero-shot image captioning methods reduce this dependence. These methods are usually trained primarily on text corpora, but require direct processing of real image input during the inference phase. This results in a significant difference in input modality between the training and inference phases. Particularly in cross-domain scenarios, the language distribution, visual content organization, and semantic expression of images in the training corpus are often inconsistent, further complicating visual-language alignment and impacting the accuracy and stability of the generated descriptions.

[0003] To alleviate the aforementioned problems, existing technologies typically enhance the semantic modeling capabilities of zero-shot image description models from the perspective of constructing auxiliary information. One type of method provides additional linguistic context for the generation process by retrieving text descriptions that are semantically related to the input; another type of method utilizes text-to-image generation models to construct synthetic images to supplement the visual input missing during the training phase. These two types of methods improve upon the limitations of text-only training from the aspects of semantic expansion and visual compensation, respectively, but certain shortcomings still exist. The information introduced by the former mainly comes from text similarity and lacks direct visual constraints. When the distribution of training corpus and test images differs significantly, the semantic associations in the retrieval results may not stably correspond to the current visual content. Although the latter can provide visual information during the training phase, the quality of the synthetic images is greatly affected by the text representation and the capabilities of the generation model, and some samples may still deviate from the original description. Therefore, relying solely on a single auxiliary method is still insufficient to simultaneously address visual compensation during the training phase, semantic support during the retrieval phase, and consistency between the training and inference processes. Summary of the Invention

[0004] The purpose of this invention is to provide a zero-shot image description method based on synthetic image retrieval, comprising: Step S100: Input the text descriptions from the training corpus into the pre-trained text-to-image generation model and output a set of synthesized images corresponding to the text descriptions; Step S200: Obtain the original text features and synthetic image features of the text description, calculate the image-text semantic similarity between the original text features and the synthetic image features, and retain the high consistency synthetic images with image-text semantic similarity higher than the similarity threshold; Step S300: Use the high-consistency synthesized image as the visual query input, and traverse the set of retrieval text features in the training corpus that are semantically related to the high-consistency synthesized image. Step S400: The original text features, the features of the highly consistent synthesized image, and the retrieved text feature set are input into the feature fusion network for unified modeling and attention update, and the output is a conditional prefix representation for language decoding; Step S500: Input the conditional prefix representation into the autoregressive language model for text decoding and generate description results, calculate the loss to optimize the network parameters, and output the trained zero-shot image description model. Step S600: Input the real image to be processed into the trained zero-sample image description model, replace the synthetic image with the real image, execute steps S300 and S400 to output the target conditional prefix representation, input the target conditional prefix representation into the autoregressive language model, and output the final text description for the real image.

[0005] Furthermore, step S300 specifically includes: Step S310: Construct a text corpus for image-to-text retrieval; Step S320: Use CLIP text encoder to encode the text in the text corpus offline to obtain the corresponding text feature set; Step S330: Extract the visual features of each synthesized image using the CLIP image encoder; Step S340: Calculate the semantic similarity between the visual features of the synthesized image and each text feature in the text corpus; Step S350: Sort the texts according to their similarity scores and select the top-ranked texts. K The text is used as the retrieval result, thereby constructing a set of retrieval text features related to the semantics of the synthesized image.

[0006] Furthermore, step S400 specifically includes: Step S410, extract the original text features Corresponding synthetic image features semantic similarity between the two Synthetic image dataset as input , n To measure the number of text descriptions in the training corpus; Step S420: Obtain sample-level cross-modal fusion feature representations. , ; Step S430: Represent the cross-modal fusion features through linear mapping. Projected onto the high-dimensional feature space of the language model and reconstructed into a fixed-length intermediate semantic sequence. , , in, Represents a linear mapping function. and For linear mapping parameters; Step S440: Analyze the feature set of the retrieved text. Perform a linear mapping to obtain ,make With intermediate semantic sequence Located in the same feature space, , in, Represents a linear mapping function. and For linear mapping parameters; Step S450, with As a query, the retrieved text features are used as keys and values. The semantic representation is updated through cross-attention to obtain a representation that integrates the retrieval semantics: , in, This is for cross-attention operations; Step S460, introduce a set of lengths of p Learnable prefix vector sequence C and combine it with the updated semantic representation The sequences are concatenated to form the input sequence for the prefix generation module. The model is then input into a Transformer for modeling, and the output at the corresponding prefix position is extracted as the final prefix representation, resulting in a prefix representation that takes into account text constraints, visual information, and retrieval semantics. , ， .

[0007] Further, step S500, which involves inputting the conditional prefix representation into the autoregressive language model for text decoding and generating descriptive results, includes: In generating the first When there are 1 word, the model uses prefix representation. and historical word sequence For the current word Modeling the conditional probability, i.e. ; Target Description The generation probability is expressed as: , Where T is the total length of the text description.

[0008] Furthermore, the loss function in step S500 is: .

[0009] Furthermore, in step S100, the text-to-image generation model is the Stable Diffusion 2.1 model.

[0010] Furthermore, in step S100, the CLIP ViT-B / 32 encoder model is used to process the text description. t i With the corresponding synthetic image s i Encode the original text features to obtain them. and synthetic image features .

[0011] Compared with the prior art, the present invention has the following technical advantages: (1) In order to reduce the difference in input form between the training stage and the inference stage, the present invention further introduces a unified image-to-text retrieval mechanism. In the training stage, synthetic images are used as visual queries, and in the inference stage, real input images are used as visual queries, thereby establishing a consistent cross-modal semantic interaction process between the two stages. (2) The model combines synthetic image features, text features and relevant semantic information obtained from retrieval to construct a conditional representation for describing the generation, and guides the language model to generate the final description through prefix mapping, which supplements the model with semantic context related to the current visual content, thereby alleviating the modality difference problem under the condition of only text training to a certain extent, and improving the model's description generation ability in cross-domain zero-sample scenarios. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0013] Figure 2 This is a schematic diagram illustrating the semantic similarity distribution of synthetic image-text pairs on the MSCOCO and Flickr30k datasets.

[0014] Figure 3This diagram illustrates the comparison of retrieval performance under different data sources and encoder structures. (a) shows the retrieval results using the ViT-B / 32 encoder from different data sources, (b) shows the retrieval results using the RNx50 encoder from different data sources, (c) shows the retrieval results using the ViT-L / 14 encoder from different data sources, and (d) shows the retrieval results using synthetic images with different encoders.

[0015] Figure 4 The diagrams show the visualization results of feature distribution for different fusion methods. (a) shows the feature distribution of text, image, and synthesized image; (b) shows the feature distribution of different fusion methods; (c) shows the feature distribution of reference image and corresponding synthesized image; and (d) shows the feature distribution of forward fusion. Detailed Implementation

[0016] Combination Figure 1 A zero-shot image description method based on synthetic image retrieval, comprising: Step S100: Input the text descriptions from the training corpus into the pre-trained text-to-image generation model and output a set of synthesized images corresponding to the text descriptions; Step S200: Obtain the original text features and synthetic image features of the text description, calculate the image-text semantic similarity between the original text features and the synthetic image features, and retain the high consistency synthetic images with image-text semantic similarity higher than the similarity threshold; Step S300: Use the high-consistency synthesized image as the visual query input, and traverse the set of retrieval text features in the training corpus that are semantically related to the high-consistency synthesized image. Step S400: The original text features, the features of the highly consistent synthesized image, and the retrieved text feature set are input into the feature fusion network for unified modeling and attention update, and the output is a conditional prefix representation for language decoding; Step S500: Decode the conditional prefix representation into text and input it into the autoregressive language model to decode the text and generate description results. Calculate the loss to optimize the network parameters and output the trained zero-shot image description model. Step S600: Input the real image to be processed into the trained zero-sample image description model, replace the synthetic image with the real image, execute steps S300 and S400 to output the target conditional prefix representation, input the target conditional prefix representation into the autoregressive language model, and output the final text description for the real image.

[0017] In step S100, the text-to-image generation model is the Stable Diffusion 2.1 model, which can generate high-quality images based on the user's text description. (For text containing only...)n Training set of text descriptions This embodiment uses a pre-trained text-to-image generation model. G For each text t i Generate synthetic images s i ,Right now s i = G ( t i Based on the synthesized image s i Generate synthetic image set S .

[0018] In step S100, the goal is not to obtain a generated result whose visual quality is completely close to that of a real image, but to use the object, scene and relationship information contained in the text to construct an image representation that is as consistent as possible with the descriptive semantics, thereby providing input for subsequent visual feature extraction.

[0019] In step S200, the CLIP ViT-B / 32 encoder model is used to process the text description. t i With the corresponding synthetic image s i Encode the original text features to obtain them. and synthetic image features ; Calculate the similarity between the two in a shared semantic space The similarity between the generated image and the original text is used to measure the degree of matching. A higher similarity between the text features and the synthesized image features indicates that the synthesized image is semantically closer to the text description, and its reliability as subsequent visual input is relatively stronger; conversely, a lower similarity indicates that the generated result may deviate from the text in terms of object, scene, or relationship representation. In this embodiment, a threshold is set. Only samples with similarity higher than a threshold are retained for subsequent modeling.

[0020] In this embodiment, after obtaining synthetic images with high consistency with the text description, image-to-text retrieval is further performed using these images to supplement semantic information related to the current visual content. Compared to relying solely on synthetic image features for description generation, the retrieved text description can provide the model with additional semantic context, especially when synthetic images cannot fully express the original text content; this information can supplement subsequent generation. Simultaneously, synthetic images are used for retrieval during the training phase, while real images are used during the inference phase. Both phases share the same image-to-text retrieval format, thus maintaining consistency in the semantic interaction process. Based on this, step S300 includes: Step S310: Construct a text corpus for image-to-text retrieval. ; Step S320: Use the CLIP text encoder to encode the text in the text corpus offline to obtain the corresponding text feature set. ; Step S330: Extract each composite image using the CLIP image encoder. visual features ; Step S340, calculate the synthesized image visual features Semantic similarity with each text feature in the text corpus ; Step S350: Sort the texts according to their similarity scores and select the top-ranked texts. K The text is used as the search result, thus constructing a structure related to the image. Semantic relevance search text feature set .

[0021] After obtaining the synthetic image features, original text features, and retrieved text features, the model needs to further organize this information into a unified conditional representation for subsequent description generation. Since the consistency between the image and text varies among different samples, directly using a single modality feature in subsequent modeling can easily lead to the influence of low-consistency samples. Therefore, this embodiment uses similarity scores to weightedly fuse the synthetic image features and text features to obtain a sample-level cross-modal fusion feature representation. After obtaining the cross-modal fusion feature, this embodiment further enhances it using the text feature set obtained in the retrieval stage. Step S400 specifically includes: Step S410, extract the original text features Corresponding synthetic image features semantic similarity between the two Synthetic image dataset as input ; Step S420: Obtain sample-level cross-modal fusion feature representations. , ; Step S430: Project the cross-modal fusion feature representation onto the high-dimensional feature space of the language model through linear mapping, and reconstruct it into a fixed-length intermediate semantic sequence. : , in, Represents a linear mapping function. and For linear mapping parameters; Step S440: Analyze the feature set of the retrieved text. Perform a linear mapping to obtain ,make With intermediate semantic sequence Located in the same feature space, , in, Represents a linear mapping function. and For linear mapping parameters; Step S450, with As a query, the retrieved text features are used as keys and values. The semantic representation is updated through cross-attention to obtain a representation that integrates the retrieval semantics: , in, This is for cross-attention operations; Step S460, introduce a set of lengths of p Learnable prefix vector sequence C and combine it with the updated semantic representation The sequences are concatenated to form the input sequence for the prefix generation module. The model is then input into a Transformer for modeling, and the output at the corresponding prefix position is extracted as the final prefix representation, resulting in a prefix representation that takes into account text constraints, visual information, and retrieval semantics. , , .

[0022] In step S500, the language model is represented by a prefix. As a conditional input, an autoregressive approach is used to generate descriptive text. In generating the first... When there are 1 word, the model uses prefix representation. and historical word sequence For the current word Modeling the conditional probability, i.e. , From the perspective of the overall sequence, the target description The generation probability is expressed as: , During the training phase, cross-entropy loss is used to optimize the generated results. For one-hot target word annotation, the loss function is... ; During the inference phase, given the input image, the model generates the final description step by step using the same prefix construction and decoding method.

[0023] To verify the effectiveness of the method in this embodiment for zero-shot image description tasks, the model is evaluated in both intra-domain and cross-domain settings. The experiments use the commonly used MSCOCO and Flickr30k datasets for image description tasks, and employ BLEU, METEOR, CIDEr, and SPICE as evaluation metrics.

[0024] The MSCOCO dataset contains a large number of everyday scene images with rich object categories and complex scene semantics, making it one of the most commonly used evaluation datasets for image description tasks. Each image typically corresponds to multiple manually annotated descriptions, enabling a comprehensive evaluation of the model's ability to generate descriptions for complex visual content. In this embodiment, following the Karpathy partitioning standard, the dataset is divided into a training set (113,287 images), a validation set (5,000 images), and a test set (5,000 images), with the training set containing 566,747 text descriptions.

[0025] The Flickr30k dataset differs from MSCOCO in image content and language style, with its descriptions leaning more towards natural scenes and human activities. Therefore, it is suitable for evaluating a model's ability to generate descriptions under varying distribution conditions. Flickr30k is divided into a training set (29,000 images), a validation set (1,004 images), and a test set (1,000 images), with the training set containing 145,000 text descriptions.

[0026] BLEU measures the surface matching of a description by statistically analyzing the degree of n-gram overlap between the generated and reference texts; this embodiment uses BLEU-4 as the evaluation metric. METEOR further considers lexical variations and synonym matching, reflecting the semantic closeness between the generated and reference descriptions to a certain extent. CIDEr assigns word frequency-based weights to different n-grams, focusing more on the expression of key information in the description, and can better reflect the overall semantic matching degree between the generated and reference descriptions in image description tasks. SPICE parses the generated and reference texts into scene graphs and matches them at the object, attribute, and relation levels, making it more suitable for evaluating the accuracy of the semantic structure in the description.

[0027] (1) Experimental setup In the image synthesis stage, this embodiment uses Stable Diffusion 2.1 as the text-to-image generation model. Under fixed parameter conditions, it generates a synthetic image with a resolution of 512×512 based on each text description, with a sampling step count of 20. In the image selection stage, CLIP ViT-B / 32 is used to encode both the text and the synthetic image, and selection is performed based on the normalized image-text similarity, with a threshold value... The value is set to 60. In the retrieval enhancement stage, this embodiment sorts the image features extracted by RNx50 with the features in the text corpus based on their similarity, and selects the top... The search results are text snippets. Different search scales are used for different datasets; for MSCOCO, the top-7 are used, and for Flickr30k, the top-9 are used. In the prefix construction stage, this embodiment linearly projects the fused features into a semantic sequence of length 10, and constructs a prefix representation by combining it with the search text features; the learnable prefix length is also set to 10. In the description generation stage, GPT-2 is used as the autoregressive language model decoder, and in the inference stage, a bundle search strategy is used with a bundle width of 5. In the overall settings, the image-text pairing data in the target dataset is not used during model training.

[0028] (2) Experimental environment The hardware and software information used in this embodiment is shown in Table 1.

[0029] Table 1 Experimental Equipment and Environment

[0030] (3) Comparison method To comprehensively evaluate the effectiveness of the method in this embodiment, representative methods with similar settings are selected as comparison objects. These methods, considering their respective technical approaches, mainly include training-free methods, text-only training methods, retrieval enhancement methods, and synthetic data augmentation methods. All metrics are derived from the corresponding original papers.

[0031] Training-free methods: ZeroCap is a typical training-free, zero-shot image captioning method. It does not rely on additional parameter updates but directly utilizes the collaborative constraints between pre-trained visual-language models and language models to generate descriptions. MAGIC further enhances the stability of the generation process through constraint optimization during the inference stage, thereby improving the semantic consistency of the description results. These methods are characterized by not requiring additional training but are highly dependent on the decoding strategy and language priors during the inference stage.

[0032] Text-only training methods: ViECap explicitly introduces visually relevant entity information during the generation process to enhance the consistency between the description content and the image semantics. CapDec achieves zero-shot image description generation by constructing continuous prefix representations during the training phase, mapping visual features to the input space of the language model. DeCap further utilizes text data to construct training objectives, completing prefix learning without relying on image-text pairing supervision, representing a significant work in text-only training methods. These methods can improve description generation capabilities through training, but due to the lack of real visual input during the training phase, their modeling of visual-language alignment still has certain limitations.

[0033] Retrieval Enhancement Methods: IFCap further enhances the generation process by combining retrieval mechanisms and supplementing visual conditions with external semantic information, demonstrating good adaptability in cross-domain scenarios. Compared to the method in this embodiment, this type of method emphasizes improving description quality by leveraging retrieval results or entity information, but it does not address the differences between text-only training and real-image reasoning from the perspective of consistency of input format during the training phase.

[0034] Synthetic data augmentation method: SynTIC constructs synthetic images using a text-to-image generation model and uses them for zero-shot image description training to alleviate the problem of missing visual input under text-only training conditions. ICSD

[54] This method also utilizes synthetic data to enhance the description of the generation process, but focuses more on the auxiliary role of synthetic images in visual conditional modeling during the training phase. This type of method is most similar to the one in this embodiment, both attempting to reduce the modal differences between the training and inference phases through synthetic images. Therefore, it can more directly reflect the improvement in the way synthetic data is utilized in the method of this embodiment.

[0035] Compared to these methods, the method in this embodiment combines synthetic image construction and image-to-text retrieval, maintaining a consistent retrieval format throughout the training and inference phases, and completing multi-source feature fusion and description generation on this basis. Subsequent experiments will further compare the method in this embodiment with the aforementioned comparative methods, focusing on both intra-domain and cross-domain settings.

[0036] (4) Analysis of experimental results First, analysis of experimental results within the domain.

[0037] The experimental results are shown in Table 2. B@4, M, C, and S represent the metrics BLEU-4, METEOR, CIDEr, and SPICE, respectively. Higher values ​​indicate better model performance; "-" indicates a missing value; bold indicates the optimal result for that metric; and underline indicates the second-best result. SRCap demonstrates strong competitiveness on both datasets, especially achieving the best result on the CIDEr metric, indicating that this method performs well in matching key information between generated descriptions and reference text.

[0038] Table 2. Experimental results of the SRCap model using zero-shot in-domain images on the MSCOCO and Flickr30k datasets.

[0039] On the MSCOCO dataset, SRCap achieves superior results across four metrics: B@4, METEOR, CIDEr, and SPICE. Compared to the training-free methods ZeroCap and MAGIC, SRCap shows a significant improvement, indicating that relying solely on inference stage constraints is insufficient to establish stable visual-language correspondences. Compared to the text-only training methods CapDec and DeCap, SRCap also maintains its lead on CIDEr and SPICE. Although there is still a significant gap in intra-domain performance compared to IFCAP, it preliminarily suggests that in complex scenes and images with multiple co-occurring objects, conditional representations learned solely from text are still insufficient, while the visual information from synthetic images helps enhance the model's semantic modeling ability of image content.

[0040] On the Flickr30k dataset, SRCap did not achieve optimal results across all metrics, but it reached 65.5 on CIDEr, the best result in the table; meanwhile, other metrics also remained at a high level. Compared to SynTIC, SRCap improved CIDEr by 8.9 percentage points; compared to ICSD, CIDEr improved by 11.2 percentage points. Furthermore, the gap with IFCAP narrowed, and SRCap even surpassed it in some aspects. These results indicate that, in data distributions like Flickr30k where image content and language expression styles are relatively free, the retrieval semantics introduced by the method in this embodiment can effectively supplement the visual information provided by the synthesized images, thereby improving the overall semantic matching degree between the generated description and the reference text. On the other hand, SRCap's SPICE index on Flickr30k is lower than SynTIC, indicating that there is still room for further optimization in the fine-grained expression of entity relationships in this embodiment.

[0041] The comparison results of different technical approaches show that synthetic data augmentation methods generally outperform training-free methods and some text-only training methods, indicating that supplementing visual information during the training phase is necessary for zero-shot image description tasks. Compared with SynTIC and ICSD, which also utilize synthetic images, SRCap's further advantages are mainly reflected in two aspects: First, this embodiment reduces the interference of biased samples on subsequent modeling by filtering and weighted fusion of synthetic images; second, by adding image-to-text retrieval, the model can obtain additional semantic information related to the current visual content during the generation phase. The combined effect of these two factors enables SRCap to achieve more stable overall performance within the domain setting.

[0042] Second, analysis of cross-domain experimental results.

[0043] Cross-domain experiments were used to evaluate the model's description generation capability under conditions where the training data distribution and the test data distribution are inconsistent. Specifically, after training on the source dataset, the model was directly transferred to the target dataset for inference. The experimental results are shown in Table 3. SRCap achieved good results in both cross-domain settings, especially maintaining a leading position in the CIDEr metric, indicating that the method in this embodiment can still effectively preserve the key information in the description even when the data distribution changes.

[0044] Under the MSCOCO→Flickr30k setting, SRCap achieved optimal results in all metrics. Compared to the training-free method MAGIC, SRCap showed a significant improvement, indicating that relying solely on inference stage constraints is insufficient to adapt to changes in image content across domain scenarios. Compared to the text-only training methods CapDec and DeCap, SRCap improved by 15.1 percentage points on CIDEr, demonstrating that when the language style and visual distribution of training and testing data change, the conditional representation obtained solely from text modality learning still has significant limitations. Compared to ViECap, SynTIC, and IFCap, SRCap still improved on the CIDEr metric by 12.4, 12.4, and 3.3 percentage points, respectively, indicating that while using retrieval augmentation alone or synthetic images alone can bring some improvement, combining the two further enhances the model's semantic matching ability in the target domain.

[0045] Table 3. Experimental results of the SRCap model on cross-domain zero-shot image descriptions on the MSCOCO and Flickr30k datasets.

[0046] In the Flickr30k→MSCOCO setting, SRCap's METEOR is tied for the highest with IFCap, while CIDEr and SPICE are the best. This indicates that when migrating from a smaller training dataset to the semantically richer and more complex MSCOCO dataset, the method in this embodiment can still maintain good description quality. Furthermore, when the target domain image content becomes more complex, relying solely on synthetic images or solely on retrieval supplementation is insufficient to adequately handle distribution changes. The method in this embodiment, by simultaneously utilizing synthetic images and image-to-text retrieval, can more stably maintain the overall semantic consistency between the description and the image content.

[0047] (5) Visualization analysis First, visualize and analyze the selection of composite images.

[0048] To analyze the semantic consistency between synthesized images and their corresponding text descriptions, this embodiment statistically analyzes the semantic similarity distribution of synthesized image-text pairs on the MSCOCO and Flickr30k datasets, such as... Figure 2As shown, the similarity between the two sets of data is mainly concentrated in the range of 60-70, indicating that most synthetic images can reflect the main semantic information in their corresponding texts well, but there are still some low-consistency samples. If these samples are used directly for subsequent training, visual signals inconsistent with the text may be introduced.

[0049] Further statistics show that when the screening threshold is 60, MSCOCO and Flickr30k retain 468,297 and 126,649 image-text pairs, respectively, accounting for 82.63% and 87.34%. However, when the threshold is increased to 80, the retention rates drop to 0.45% and 1.30%, respectively. Considering both image-text consistency and training data retention rate, this embodiment ultimately uses 60 as the screening threshold for synthetic images to remove low-quality samples while retaining as many synthetic images as possible that can be used for subsequent modeling.

[0050] Second, the visual analysis of the search results.

[0051] To further analyze the impact of different search query formats on the quality of search results, this paper compares the performance of three search input formats—text, original images, and synthetic images—on the MSCOCO dataset under different CLIP encoders. The results are as follows: Figure 3 As shown in the figure, four metrics are used to evaluate the search results: Entity Accuracy (ACC), Recall (RC), Average Number of Illusory Entities (AHC), and Deduplicated Number of Illusory Entities (DHC). ACC and RC measure the coverage of image-related semantics by the search results, while AHC and DHC reflect the degree to which erroneous entities are introduced during the search process. For the first two metrics, higher values ​​indicate better search results; for the latter two metrics, lower values ​​indicate fewer erroneous entities introduced during the search process.

[0052] from Figure 3 (a) to Figure 3(c) It can be seen that different retrieval query formats exhibit significant differences in results. When using the original image as the query, ACC is usually high, while RC is relatively low, and AHC and DHC also remain at low levels. This indicates that real image retrieval can more accurately constrain candidate semantics, but its returned results are relatively conservative and do not adequately cover some relevant semantics. When using text as the query, RC is often high, but ACC decreases relatively, and AHC and DHC increase significantly, indicating that relying solely on text similarity can expand the range of candidate semantics, but it is also more likely to bring in entities inconsistent with the current image content. In contrast, when using synthetic images as the query, the overall indicators are more balanced: its ACC, AHC, and DHC are significantly better than text retrieval, while its RC is higher than that of original image retrieval. This proves that synthetic images can simultaneously retain certain visual constraints and high semantic coverage during the retrieval stage, and are therefore more suitable as a retrieval query format for the training stage.

[0053] Figure 3 (d) The impact of different encoders on retrieval results under synthetic image retrieval conditions was further compared. Overall, the differences between different encoders in ACC and RC were relatively limited, indicating that synthetic images have a certain degree of stability when used as retrieval queries; however, differences still exist in AHC and DHC. Compared to ViT-B / 32 and ViT-L / 14, RNx50 performed lower in the illusion index, indicating that it introduced relatively fewer erroneous entities in its returned results. Considering the characteristics of different coding structures, this phenomenon may be related to the fact that RNx50 focuses more on local salient patterns, while the ViT series, due to its emphasis on global relation modeling, is more likely to retain some marginally related but inaccurate entities while expanding the candidate semantic range. Further comparison between ViT-B / 32 and ViT-L / 14 shows that ViT-L / 14 is slightly better than ViT-B / 32, indicating that stronger global representation capabilities still contribute to retrieval quality.

[0054] In summary, synthetic image retrieval demonstrates a better balance between semantic coverage and error control, providing a more suitable image-to-text retrieval query format for the training phase. The RNx50 encoder, while maintaining retrieval effectiveness, exhibits lower overall HC and DHC metrics. Therefore, this paper selects it as the default retrieval encoder for subsequent experiments.

[0055] Third, visualization analysis of the fusion results.

[0056] from Figure 4(a) It can be seen that there is a significant difference in distribution between text features and visually relevant features. Text features are generally concentrated on one side of the representation space, while synthetic image features and image features are mainly distributed on the other side, with some overlap between them. This indicates that when relying solely on text features, there is still a significant modal difference between the representation learned by the model and the real visual input. In contrast, synthetic image features have a distribution that is closer to that of image features, and can supplement the visual information missing under text-only training conditions to some extent, but there is still a difference between their distribution and that of real image features.

[0057] Figure 4 (b) further illustrates the feature distribution relationship between the reference image and the corresponding synthesized image. It can be seen that some synthesized image features are distributed near the reference image features, indicating that these samples can preserve the original semantic information relatively well; however, at the same time, there are still some synthesized image features that deviate significantly from the reference image features. This suggests that although synthesized images can provide visual supplementation, their semantic quality is not stable, and if a single synthesized image feature is directly used in subsequent modeling, it may still be affected by generation bias.

[0058] Figure 4 (c) The trends in feature distribution under different fusion directions and fusion coefficients were compared. Positive and negative fusion were dynamically weighted based on semantic similarity, while fusion using different fusion coefficients was a statically weighted fusion with fixed weights. It can be seen that when the fused features retain more textual information, their distribution is still significantly biased towards the textual feature region; as the proportion of visual information increases, the fused features gradually shift towards the regions containing image features and reference image features. Specifically, the features obtained by positive fusion and larger fusion coefficients are generally closer to the feature distribution of the reference image, while the features obtained by negative fusion and smaller fusion coefficients are more biased towards the text side. This result indicates that the fusion direction and fusion weights directly affect the position of the fused representation in the semantic space; reasonable fusion settings can more fully absorb the visual information in the synthesized image while maintaining the semantic constraints of the text.

[0059] Figure 4 (d) presents the final distribution of the forward fusion features. Compared to text features, the fusion features shift significantly towards the visual feature region; compared to single synthesized image features, the fusion features are closer to the location of the reference image features. This indicates that the final fusion representation does not remain on one side of a single modality, but rather forms a more balanced representation between textual semantics and visual information.

[0060] In summary, text features, synthetic image features, and real image features differ significantly in their representation spaces, making it difficult for a single modality representation to simultaneously satisfy semantic constraints and visual consistency. Appropriate feature fusion can unify the semantic information in the text with the visual information in the synthetic image, making the fused features more closely resemble the feature distribution of the reference image.

Claims

1. A zero-shot image description method based on synthetic image retrieval, characterized in that, include: Step S100: Input the text descriptions from the training corpus into the pre-trained text-to-image generation model and output a set of synthesized images corresponding to the text descriptions; Step S200: Obtain the original text features and synthetic image features of the text description, calculate the image-text semantic similarity between the original text features and the synthetic image features, and retain the high consistency synthetic images with image-text semantic similarity higher than the similarity threshold; Step S300: Use the high-consistency synthesized image as the visual query input, and traverse the set of retrieval text features in the training corpus that are semantically related to the high-consistency synthesized image. Step S400: The original text features, the features of the highly consistent synthesized image, and the retrieved text feature set are input into the feature fusion network for unified modeling and attention update, and the output is a conditional prefix representation for language decoding; Step S500: Input the conditional prefix representation into the autoregressive language model for text decoding and generate description results, calculate the loss to optimize the network parameters, and output the trained zero-shot image description model. Step S600: Input the real image to be processed into the trained zero-sample image description model, replace the synthetic image with the real image, execute steps S300 and S400 to output the target conditional prefix representation, input the target conditional prefix representation into the autoregressive language model, and output the final text description for the real image.

2. The method according to claim 1, characterized in that, Step S300 specifically includes: Step S310: Construct a text corpus for image-to-text retrieval; Step S320: Use CLIP text encoder to encode the text in the text corpus offline to obtain the corresponding text feature set; Step S330: Extract the visual features of each synthesized image using the CLIP image encoder; Step S340: Calculate the semantic similarity between the visual features of the synthesized image and each text feature in the text corpus; Step S350: Sort the texts according to their similarity scores and select the top-ranked texts. K The text is used as the retrieval result, thereby constructing a set of retrieval text features related to the semantics of the synthesized image.

3. The method according to claim 2, characterized in that, Step S400 specifically includes: Step S410, extract the original text features Corresponding synthetic image features semantic similarity between the two Synthetic image dataset as input , n To measure the number of text descriptions in the training corpus; Step S420: Obtain sample-level cross-modal fusion feature representations. , ； Step S430: Represent the cross-modal fusion features through linear mapping. Projected onto the high-dimensional feature space of the language model and reconstructed into a fixed-length intermediate semantic sequence. , ， in, Represents a linear mapping function. and For linear mapping parameters; Step S440: Analyze the feature set of the retrieved text. Perform a linear mapping to obtain ,make With intermediate semantic sequence Located in the same feature space, ， in, Represents a linear mapping function. and For linear mapping parameters; Step S450, with As a query, the retrieved text features are used as keys and values. The semantic representation is updated through cross-attention to obtain a representation that integrates the retrieval semantics: ， in, This is for cross-attention operations; Step S460, introduce a set of lengths of p Learnable prefix vector sequence C and combine it with the updated semantic representation The sequences are concatenated to form the input sequence for the prefix generation module. The model is then input into a Transformer for modeling, and the output at the corresponding prefix position is extracted as the final prefix representation, resulting in a prefix representation that takes into account text constraints, visual information, and retrieval semantics. , ， 。 4. The method according to claim 3, characterized in that, Step S500, which inputs the conditional prefix representation into the autoregressive language model for text decoding and generates descriptive results, includes: In generating the first When there are 1 word, the model uses prefix representation. and historical word sequence For the current word Modeling the conditional probability, i.e. ； Target Description The generation probability is expressed as: ， Where T is the total length of the text description.

5. The method according to claim 4, characterized in that, The loss function in step S500 is: 。 6. The method according to claim 1, characterized in that, In step S100, the text-to-image generation model is the StableDiffusion 2.1 model.

7. The method according to claim 1, characterized in that, In step S100, the CLIP ViT-B / 32 encoder model is used to process the text description. t i With the corresponding synthetic image s i Encode the original text features to obtain them. and synthetic image features .

Images