Image caption generation method and system based on scene graph and entity-driven prompt

Abstract

The present application relates to a scene graph and entity-driven prompt-based image caption generation method and system, including generating an image scene graph and extracting core entities, and coarsely screening candidate captions based on the core entities; converting the coarsely screened captions into a language scene graph, and finely screening through shared GCN coding and structured matching; combining the fine screening results and the core entities to construct soft and hard prompts, and inputting the self-attention model after cross-attention collaboration to generate captions. The present application realizes fine-grained cross-modal alignment through a scene graph, and fuses key information through weighted soft and hard prompts, and outperforms the baseline model on the COCO Test and NoCaps Val data sets, improving the accuracy and generalization ability of image caption generation, and being suitable for image description tasks in the cross field of computer vision and natural language processing. The present application solves the problems of key entity omission and topological relationship mismatch caused by coarse-grained alignment in existing image-text bimodal retrieval, and the problem of insufficient fine-grained entity perception caused by insufficient fusion of soft and hard prompts in lightweight models.

Description

Technical Field

[0001] This invention relates to the fields of computer science and artificial intelligence, and in particular to a method and system for generating image captions based on scene graphs and entity-driven prompts. Background Technology

[0002] Image captioning, a core task at the intersection of computer vision and natural language processing, aims to transform visual images into semantically coherent text descriptions that conform to human language habits. It has irreplaceable application value in areas such as intelligent visual interaction (e.g., accessibility assistance, intelligent photo album annotation), autonomous driving scene understanding, and cross-modal content retrieval. Statista data shows that the global computer vision AI market exceeded $50 billion in 2024, with the commercial penetration rate of image captioning-related technologies growing at an annual rate of over 25%, demonstrating its technological importance and market potential. However, existing technologies still have significant shortcomings in addressing issues such as the omission of key entities and mismatches in topological relationships due to coarse-grained retrieval, and insufficient fine-grained entity perception capabilities due to inadequate fusion of hardware and software cues, making it difficult to meet the generation needs of complex scenarios.

[0003] Early image captioning generation methods largely relied on retrieval-based approaches. These methods involved pre-building caption libraries, matching the input image with features from these libraries, and directly reusing captions from similar images. The BabyTalk method proposed by Kulkarni et al. first used a detector to identify visual elements, then retrieved and filled in the optimal sentence structure from a predefined database based on a Conditional Random Field (CRF). While this achieved preliminary structured retrieval, it relied solely on explicit template matching and lacked generalization ability in open scenes. Farhadi et al. constructed an "object-action-scene" triplet retrieval system, generating descriptions by retrieving the most similar training samples. However, this approach remained limited to predefined semantic slots and failed to capture the dynamic relationships between entities.

[0004] With the development of deep learning, the encoder-decoder framework has become mainstream. Vinyals et al. first proposed an end-to-end model (Show and Tell) using CNNs (such as AlexNet) as image encoders and RNNs as text decoders, achieving a BLEU-4 performance of 27.5 on the MSCOCO dataset, laying the technological foundation for modern image captioning generation. However, this framework lacks attention to local image regions, easily generating "template-based" captions (such as "a dog is on the grass"), and failing to distinguish entity details. To address this issue, Xu et al. introduced a "visual attention mechanism" (Show, Attend, and Tell), enabling the decoder to dynamically focus on key image regions when generating text, improving the BLEU-4 score on the MSCOCO dataset to 34.7. However, this method is still limited to "independent encoding of single-modal features" and does not achieve deep semantic alignment between images and text.

[0005] In recent years, vision-language pre-trained (VLP) models have further advanced cross-modal alignment capabilities. Zhou et al. proposed a unified vision-language pre-training framework, introducing vision-word level attention within a Transformer structure to achieve deep fusion encoding of image region features and text word embeddings. Li et al.'s OSCAR model innovatively introduces object labels generated by an object detector as visual "anchors," significantly improving the ability to capture key objects by explicitly strengthening the alignment between visual entities and language segments. Zhang et al.'s VinVL model performs secondary pre-training based on stronger object detection features, further optimizing fine-grained region-word joint representations.

[0006] In the field of Retrieval Augmentation (RAG), existing methods generally rely on matching patterns of "image global vector - text vector" (such as CLIP and BLIP series). These methods can only capture global semantic similarity and cannot distinguish entity attributes and relationships. For example, for images containing "parking meter, truck, car", Wang et al.'s cross-modal retrieval mechanism can retrieve candidate captions containing "parking meter", but because it relies on global feature similarity, it cannot distinguish the relationship difference between "parking meter next to truck" and "truck next to parking meter", resulting in a retrieval error rate of over 25%.

[0007] While some studies have introduced scene graphs to optimize the internal attention of models (such as Yao et al. using GCN to model the interaction relationships between objects and improve the logical coherence of generated sentences), they have not used it as a "structured query" for retrieval. Hassani et al. pointed out in their research on graph contrastive learning that "vector-level matching" cannot utilize the topological information of the graph structure, while the "entity-relationship" structure of the scene graph is precisely the key to distinguishing semantic differences. This technological gap makes it difficult to improve the accuracy of current retrieval. For example, Cornia et al.'s Meshed-Memory Transformer, although constructing a hierarchical external memory bank, still did not introduce structured topological constraints, and the problem of relationship confusion still exists in multi-entity interaction scenarios. Summary of the Invention

[0008] The purpose of this invention is to provide an image caption generation method and system based on scene graph and entity-driven prompts, which can achieve fine-grained cross-modal matching and structured prompt fusion, thereby improving the accuracy, generalization and robustness of caption generation.

[0009] To achieve the above objectives, the present invention provides the following technical solution: A method for generating image captions based on scene graphs and entity-driven prompts includes: Step S1: Obtain the input image, generate an image scene graph, and extract the core entities of the image; It is a triplet relation; Step S2: Based on the global CLIP feature vector of the image, the candidate caption set is coarsely filtered by the FAISS index to obtain a coarsely filtered candidate caption subset; Step S3: The coarsely screened candidate subtitle subset is transformed into a language scene graph set. The image scene graph and the language scene graph are encoded by the shared graph convolutional network GCN. The cross-modal semantic matching score and the structured matching score are calculated respectively. The inverse ranking fusion RRF strategy is used to combine the scores and perform fine screening to output the top m candidate subtitles. Step S4: Based on the core entity of the image and the first m candidate subtitles, construct hard cues and soft cues respectively to form a soft and hard cue collaborative input; Step S5: Input the soft and hard prompts together into the autoregressive generation model, and inject the core entity features into the deep network of the generation model through the deep injection mechanism to output image captions.

[0010] Furthermore, the image scene graph in S1 is defined as: ; This is a triplet relation. This represents the original visual scene graph constructed from the input image. This represents the initial set of visual vertices containing physical entities in the image, identified by the object detector. This represents the set of directed relation triples defined on the vertex set to represent the topological interaction logic between objects.

[0011] Furthermore, the method for extracting the core entity of the image in step S1 is as follows: Using the UniDet target detector from The confidence score (v) of the entity is obtained. First, the image entities are sorted in descending order by score (v). Then, the core entities are filtered through a two-layer deduplication strategy. ,satisfy: ={v1,v2,…,vk}, where k≤5, vi are the top k entities that pass the deduplication verification after sorting, and satisfy: The names of entities after standardization processing, including conversion to lowercase, removal of punctuation marks, and removal of leading and trailing whitespace, are not duplicated; The word set of an entity is not included in the word set of the selected entities; If no valid entity is found after filtering, the placeholder "(some objects)" will be used as the core entity.

[0012] Furthermore, step S2 further comprises: Step A1, Image Feature Encoding: The CLIP model is used to globally encode the input image to obtain the global feature vector of the image. ∈ ,in, Let D represent the real number field space, where D is the CLIP feature dimension, and... Perform L2 normalization; Step A2: Normalize the Input the FAISS index, which is built offline based on an external large-scale image and text dataset, and retrieve the Top-K1 candidate captions that are most similar to the image features, where K1=15; Step A3, Anti-cheating Filtering: Based on the "candidate caption-image ID" mapping relationship constructed by splitting the Karpathy dataset, candidate captions that are the same image as the input image are filtered to obtain a coarsely filtered subset. .

[0013] Furthermore, step S3 further includes: Step B1: Coarsely screen candidate subtitles Constructing a language scene graph ,in A collection of text entities For syntactic relation triples between text entities; Step B2, Image scene graph node features: for For each entity in the image, the bounding box region corresponding to it is used to extract the region of interest (ROI) sub-image from the original image based on its bounding box coordinates. The CLIP model is then used to encode the region features to obtain the image node features. , ; Let D represent the real number field space, and let D represent the dimension of the feature vector. Language scene graph node features: (The rest of the text appears to be a list of features and their relationships.) The text name of each entity is encoded, and the text features are encoded using the CLIP model to obtain language node features. , ; Step B3: Construct an adjacency matrix with self-loops: ,in The original adjacency matrix, It is the identity matrix; Symmetric normalization: Calculating the adjacency matrix degree matrix The normalized adjacency matrix is ​​obtained as follows: Message Passing and Aggregation: Performing Graph Convolution Message Passing Operations ,in The image node feature matrix is ​​used; max pooling is performed on the aggregated node features to obtain the global embedding of the image scene graph. ; Step B4, Language Scene Graph Encoding: Using a GCN network that is completely shared with the image scene graph, perform the same normalization, message passing, and pooling operations as in steps B2-B3 to obtain the global embedding of the language scene graph. The specific formula is as follows: ; in Represents the identity matrix. This represents an adjacency matrix with added self-loop mechanism. The original adjacency matrix of the language scene graph. for The corresponding degree matrix, This represents the Laplacian matrix after symmetric normalization, which is the initial language node feature matrix. This represents the hidden feature matrix of the language nodes after being updated by the graph convolutional layer. This indicates that maximum value filtering is performed at the node level. This represents the global embedding vector of the language scene graph obtained by aggregating through this max pooling operation; Step B5, Cross-modal semantic matching score: ; in This represents a subset of object-type entities selected from the text entity set. type(v')=object indicates that the semantic category attribute of entity v' is determined to be an object rather than an attribute or relation. Let cosine similarity function be defined as follows: ; Step B6, Structured Matching Score: ; Step B7, RRF fusion and comprehensive score: in For smoothing parameters, They are respectively Rank after sorting in descending order; Step B8, Detailed Screening: (Please click...) The top-m candidate subtitles are filtered in descending order, where m=4, to obtain a finely filtered subset. .

[0014] Furthermore, step S4 further comprises: Step C1, Hard hints construction: ; in, These are the first m candidate subtitles output in step 3. For the core entity set of the image; Step C2, Soft suggestion construction: Core entity text encoding: using a frozen CLIP model. Text encoding is performed to obtain cross-modal entity features, using the following formula: ; in For the number of core entities, the CLIP model weights are frozen and do not participate in training; they are only used as feature extractors. Cross-modal feature projection: Constructing a projection module The 512-dimensional CLIP entity features are mapped to the latent space of the decoder using the following formula: in Let be the dimension of the decoder's hidden layer. This represents the layer normalization operation performed on the input features. This represents a fully connected layer that maps the feature dimension from 512 to H. Represents the activation function of the Gaussian error linear unit. This represents a fully connected layer whose dimensions remain unchanged. This represents the final layer normalization operation performed on the output features. This indicates the function composition operation. This indicates embedding the input entity. The projection module is the only trainable soft cue-related component, which maps the connection module to the decoder feature space. Hierarchical entity feature allocation: based on the number of decoder injection layers Evenly distributed, a hierarchical soft cue feature set is obtained: ; in For the number of injection layers, (B represents the batch size) The number of entities allocated to the i-th level), the allocation rule is as follows: If the number of entities is insufficient, it will be replenished cyclically; Deep injection conditions: For the decoder's first... A layer will perform soft hint injection if the following conditions are met: ; Where S=3 is the injection initiation layer. For injection layer spacing, For modulo operation, For hierarchical feature indexing; Soft cue residual construction: Broadcastable residuals are constructed based on the hierarchical feature dimensions, and soft cues are fused with the decoder's hidden state. The formula is as follows: ; in This represents the feature retrieved from the hierarchical feature stack and assigned to the current injection layer. Specific feature blocks; To increase the sequence length dimension, This is an operation to take the mean along the entity dimension, where This means that the number of dimensions of the tensor remains unchanged when calculating the mean (to prevent dimensional collapse) in order to support subsequent broadcast addition operations with the hidden state; Soft hint deep injection execution: The residual form of the soft hint is combined with the decoder's... Broadcasting addition in the hidden state of the layer enables soft hint injection, with the following formula: ; in For decoder number Hidden layer state, The sequence length; Beam Search Batch Adaptation: When beam search is used during the inference phase, resulting in changes in batch size, the hierarchical soft cue features are expanded to match the batch size. The formula is as follows: ; in This is the original batch size. This is the current batch size. This is an expanded layered soft cue feature; Step C3, Implementation of hardware and software prompts in tandem: Hard cue embedding encoding: Convert hard cues into a token sequence and encode them into an embedding vector, using the following formula: ; in For hard hint token sequence, For hard-coded length, For the word embedding layer of the autoregressive model; Attention mask correction: The attention mask is converted into a 4-dimensional tensor to fit the decoder's attention calculation. The formula is as follows: ; in For the original attention mask tensor; Indicates the number of dimensions of the tensor obtained; This means that when the input is 2-dimensional, two new dimensions of size 1 are inserted in the middle through slicing operations to expand it into a 4-dimensional tensor; This indicates that when the input is 3D, in The operation of inserting a new dimension of size 1 at index 2 of a tensor; Collaborative input feature construction: This involves fusing soft prompts, hard prompts, and text prefix embeddings, using the following formula: ; Where [;] represents a tensor concatenation operation along the sequence length dimension; This represents the concatenated complete input sequence embedding; This indicates that the feature matrix belongs to the real number space, and its dimension is (hard hint: length). + prefix length Decoder hidden layer dimension ; (·) denotes a learnable linear projection layer; This represents the image features after projection adaptation; This indicates that the feature matrix belongs to the real number space, and its dimension is equal to the length of the image feature sequence. x Decoder Hidden Layer Dimension ; (Q, K, V) represents a multi-head attention mechanism used to compute cross-modal interactions between the text query vector Q and the image key vectors K and V; This represents the output features of the cross-modal attention layer; This represents the fused features that are finally input to the decoder; the "+" in the formula represents the residual connection operation, which is to add the feature vectors from different sources element by element. Embed the target subtitle prefix. For prefix length, For CLIP image global features, To provide soft cue residual features, we perform collaborative interaction between soft and hard cues and image features.

[0015] Furthermore, step S5 further comprises: Step D1: Generate model input construction: ; in This refers to the operation of constructing a soft cue prefix sequence from the projected entity features. To represent converting a text string into a discrete representation that the model can recognize. Sequence segmentation operations, This represents the discrete hard hint instruction template used for bootstrapping. This represents the concatenation operation of tensors along the sequence dimension; Step D2, Optimization Objective: Employ an improved autoregressive generative model. The optimization objective is: ; in To generate a subtitle sequence, This represents the operation of finding the parameter variable that maximizes the probability function value (i.e., the optimal sequence). This represents the complete input sequence of the generated model, including soft prompt prefixes and hard prompt instructions, after concatenation. For sequence length, The core entity features after projection; Step D3, Word-by-Word Reasoning: The reasoning stage generates subtitles through word-by-word prediction. The formula is: ; in For the t-th generated word, For the front One generated word.

[0016] Furthermore, the improved autoregressive generative model is compatible with the XGLM / OPT architecture, and the soft prompting implementation logic is consistent with GPT2, requiring only adaptation to the layer structure and embedding dimension of the corresponding decoder.

[0017] A scene graph retrieval and soft / hardware prompting collaborative image caption generation system includes a memory and a processor, capable of implementing the method steps.

[0018] The beneficial effects of this invention are as follows: Fine-grained cross-modal alignment: By modeling the "entity-relationship" structure through scene graphs, the traditional high-dimensional vector matching is replaced, improving the entity matching accuracy of the COCO Test dataset by 1%-1.5%; Structured prompt fusion: Both soft and hard prompts contain core entities and highly relevant captions, and the soft prompts are weighted according to weights, resulting in strong generalization ability: On the NoCapsVal dataset (containing 400 unseen entity categories), the CIder metric is 3 percentage points better than SmallCap (7M parameters); High reproducibility: All modules are implemented based on open-source tools, with clear parameter settings, facilitating engineering implementation. Attached Figure Description

[0019] Figure 1 This is a flowchart of the method of the present invention; Detailed Implementation

[0020] The invention will now be further described with reference to the accompanying drawings.

[0021] Please see Figure 1 The present invention provides an embodiment: a method for generating image captions based on scene graphs and entity-driven prompts, comprising: Step S1: Obtain the input image, generate an image scene graph, and extract the core entities of the image; Step S2: Based on the global CLIP feature vector of the image, the candidate caption set is coarsely filtered by the FAISS index to obtain a coarsely filtered candidate caption subset; Step S3: The coarsely selected candidate caption subset is transformed into a language scene graph set. The image scene graph and the language scene graph are encoded by a shared graph convolutional network (GCN). Cross-modal semantic matching score and structured matching score are calculated respectively. The inverse ranking fusion RRF strategy is used to combine the scores and perform fine screening to output the top m candidate captions. Specifically, the calculation of cross-modal semantic matching score and structured matching score is carried out by using the maximum cosine similarity of the node-level semantics between the core entities of the image and the text entities to calculate the cross-modal semantic matching score, and using the cosine similarity of the global topological feature vectors of the image and the text to calculate the structured matching score.

[0022] Step S4: Based on the image core entity and the first m candidate captions, construct hard cues and soft cues respectively to form a soft-hard cue collaborative input; wherein the construction of hard cues and soft cues specifically involves: using the first m candidate captions as the global context and the image core entity as the local constraint, and concatenating them with a preset instruction template to construct hard cues; at the same time, using a pre-trained text encoder to map the image core entity into a continuous vector representation and performing dimensional adaptation through a nonlinear projection layer to construct soft cues.

[0023] Step S5: Input the soft and hard prompts together into the autoregressive generation model, and inject the core entity features into the deep network of the generation model through the deep injection mechanism to output image captions.

[0024] Please continue reading. Figure 1 As shown, in one embodiment of the present invention, the image scene map in S1 is defined as: ; This is a triplet relation. This represents the original visual scene graph constructed from the input image. This represents the initial set of visual vertices containing physical entities in the image, identified by the object detector. This represents the set of directed relation triples defined on the vertex set to represent the topological interaction logic between objects.

[0025] Please continue reading. Figure 1 As shown, in one embodiment of the present invention, the method for extracting the core entity of the image in step S1 is as follows: Using the UniDet target detector from The confidence score (v) of the entity is obtained. First, the image entities are sorted in descending order by score (v). Then, the core entities are filtered through a two-layer deduplication strategy. ,satisfy: ={v1,v2,…,vk}, where k≤5, vi are the top k entities that pass the deduplication verification after sorting, and satisfy: The names of entities after standardization processing, including conversion to lowercase, removal of punctuation marks, and removal of leading and trailing whitespace, are not duplicated; The word set of an entity is not included in the word set of the selected entities (to avoid semantic redundancy); If no valid entity is found after filtering, the placeholder "(some objects)" will be used as the core entity.

[0026] Please continue reading. Figure 1 As shown, in one embodiment of the present invention, step S2 further comprises: Step A1, Image Feature Encoding: The CLIP model is used to globally encode the input image to obtain the global feature vector of the image. ∈ ,in, Let D represent the real number field space, where D is the CLIP feature dimension, and... Perform L2 normalization; Step A2: Normalize the The input is a FAISS index built offline based on a large-scale external image and text dataset. The input is used to retrieve the Top-K1 candidate captions that are most similar to the image features, where K1=15. The construction of the FAISS index includes: extracting the global feature vectors of all image description texts in the external corpus and performing L2 normalization, and using the FAISS library to build a high-dimensional vector similarity retrieval engine. Step A3, Anti-cheating Filtering: Based on the "candidate caption-image ID" mapping relationship constructed by splitting the Karpathy dataset, candidate captions that are the same image as the input image are filtered to obtain a coarsely filtered subset. .

[0027] Please continue reading. Figure 1 As shown, in one embodiment of the present invention, step S3 further comprises: Step B1: Coarsely screen candidate subtitles Constructing a language scene graph ,in A collection of text entities For syntactic relation triples between text entities; Step B2, Image scene graph node features: for For each entity in the image, the bounding box region corresponding to it is used to extract a region of interest (ROI) sub-image from the original image based on its bounding box coordinates. The CLIP model is then used to encode the region features, resulting in image node features. , ; Let D represent the real number field space, and let D represent the dimension of the feature vector. Language scene graph node features: (The rest of the text appears to be incomplete and requires further context.) The text name of each entity is encoded, that is, the pre-trained CLIP model's text encoder maps it into a fixed-dimensional continuous vector representation. The CLIP model is then used to encode text features to obtain language node features. , ; Step B3: Construct an adjacency matrix with self-loops: ,in The original adjacency matrix, It is the identity matrix; Symmetric normalization: Calculating the adjacency matrix degree matrix The normalized adjacency matrix is ​​obtained as follows: ; Message Passing and Aggregation: Performing Graph Convolution Message Passing Operations ,in The image node feature matrix is ​​used; max pooling is performed on the aggregated node features to obtain the global embedding of the image scene graph. ; Step B4, Language Scene Graph Encoding: Using a GCN network that is completely shared with the image scene graph, perform the same normalization, message passing, and pooling operations as in steps B2-B3 to obtain the global embedding of the language scene graph. The specific formula is as follows: ; in Represents the identity matrix. This represents an adjacency matrix with added self-loop mechanism. The original adjacency matrix of the language scene graph. for The corresponding degree matrix, This represents the Laplace matrix after symmetric normalization. Initial language node feature matrix, This represents the hidden feature matrix of the language nodes after being updated by the graph convolutional layer. This indicates that maximum value filtering is performed at the node level. This represents the global embedding vector of the language scene graph obtained by aggregating through this max pooling operation; Step B5, Cross-modal semantic matching score: ; in This represents a subset of object-type entities selected from the text entity set. type(v')=object indicates that the semantic category attribute of entity v' is determined to be an object rather than an attribute or relation. Let cosine similarity function be defined as follows: ; Step B6, Structured Matching Score: ; Step B7, RRF fusion and comprehensive score: in For smoothing parameters, They are respectively Rank after sorting in descending order; Step B8, Detailed Screening: (Please click...) The top-m candidate subtitles are filtered in descending order, where m=4, to obtain a finely filtered subset. .

[0028] Please continue reading. Figure 1 As shown, in one embodiment of the present invention, step S4 further comprises: Step C1, Hard hints construction: ; in, These are the first m candidate subtitles output in step 3. For the core entity set of the image; Step C2, Soft suggestion construction: Core entity text encoding: using a frozen CLIP model. Text encoding is performed to obtain cross-modal entity features, using the following formula: ; in For the number of core entities, the CLIP model weights are frozen and do not participate in training; they are only used as feature extractors. Cross-modal feature projection: Constructing a projection module The 512-dimensional CLIP entity features are mapped to the latent space of the decoder using the following formula: in For the decoder hidden layer dimension, This represents the layer normalization operation performed on the input features. This represents a fully connected layer that maps the feature dimension from 512 to H. Represents the activation function of the Gaussian error linear unit. This represents a fully connected layer whose dimensions remain unchanged. This represents the final layer normalization operation performed on the output features. This indicates the function composition operation. This indicates embedding the input entity. The projection module is the only trainable soft cue-related component, which maps the connection module to the decoder feature space. Hierarchical entity feature allocation: based on the number of decoder injection layers Evenly distributed, a hierarchical soft cue feature set is obtained: ; in For the number of injection layers, (B represents the batch size) The number of entities allocated to the i-th level), the allocation rule is as follows: If the number of entities is insufficient, it will be replenished cyclically; Deep injection conditions: For the decoder's first... A layer will perform soft hint injection if the following conditions are met: ; Where S=3 is the injection initiation layer. For injection layer spacing, For modulo operation, For hierarchical feature indexing; Soft cue residual construction: Broadcastable residuals are constructed based on the hierarchical feature dimensions, and soft cues are fused with the decoder's hidden state. The formula is as follows: ; in This represents the feature retrieved from the hierarchical feature stack and assigned to the current injection layer. Specific feature blocks; To increase the sequence length dimension, This is an operation to take the mean along the entity dimension, where This means that the number of dimensions of the tensor remains unchanged when calculating the mean (to prevent dimensional collapse) in order to support subsequent broadcast addition operations with the hidden state; Soft hint deep injection execution: The residual form of the soft hint is combined with the decoder's... Broadcasting addition in the hidden state of the layer enables soft hint injection, with the following formula: ; in For decoder number Hidden layer state, The sequence length; Beam Search Batch Adaptation: When beam search is used during the inference phase, resulting in changes in batch size, the hierarchical soft cue features are expanded to match the batch size. The formula is as follows: ; in This is the original batch size. This is the current batch size. This is an expanded layered soft cue feature; Step C3, Implementation of hardware and software prompts in tandem: Hard cue embedding encoding: Convert hard cues into a token sequence and encode them into an embedding vector, using the following formula: ; in For hard hint token sequence, For hard-coded length, For the word embedding layer of the autoregressive model; Attention mask correction: The attention mask is converted into a 4-dimensional tensor to fit the decoder's attention calculation. The formula is as follows: ; in For the original attention mask tensor; Indicates the number of dimensions of the tensor obtained; This means that when the input is 2-dimensional, two new dimensions of size 1 are inserted in the middle through slicing operations to expand it into a 4-dimensional tensor; This indicates that when the input is 3D, in The operation of inserting a new dimension of size 1 at index 2 of a tensor; Collaborative input feature construction: This involves fusing soft prompts, hard prompts, and text prefix embeddings, using the following formula: ; Where [;] represents a tensor concatenation operation along the sequence length dimension; This represents the concatenated complete input sequence embedding; This indicates that the feature matrix belongs to the real number space, and its dimension is (hard hint: length). + prefix length Decoder hidden layer dimension ; (·) denotes a learnable linear projection layer; This represents the image features after projection adaptation; This indicates that the feature matrix belongs to the real number space, and its dimension is equal to the length of the image feature sequence. x Decoder Hidden Layer Dimension ; (Q, K, V) represents a multi-head attention mechanism used to compute cross-modal interactions between the text query vector Q and the image key vectors K and V; This represents the output features of the cross-modal attention layer; This represents the fused features that are finally input to the decoder; the "+" in the formula represents the residual connection operation, which is to add the feature vectors from different sources element by element. Embed the target subtitle prefix. For prefix length, For CLIP image global features, To provide soft cue residual features, we perform collaborative interaction between soft and hard cues and image features.

[0029] Please continue reading. Figure 1 As shown, in one embodiment of the present invention, step S5 further comprises: Step D1: Generate model input construction: ; in This refers to the operation of constructing a soft cue prefix sequence from the projected entity features. To represent converting a text string into a discrete representation that the model can recognize. Sequence segmentation operations, This represents the discrete hard hint instruction template used for bootstrapping. This represents the concatenation operation of tensors along the sequence dimension; Step D2, Optimization Objective: Employ an improved autoregressive generative model. The optimization objective is: ; in To generate a subtitle sequence, This represents the operation of finding the parameter variable that maximizes the probability function value (i.e., the optimal sequence). This represents the complete input sequence of the generated model, including soft prompt prefixes and hard prompt instructions, after concatenation. For sequence length, The core entity features after projection; Step D3, Word-by-Word Reasoning: The reasoning stage generates subtitles through word-by-word prediction. The formula is: ; in For the t-th generated word, For the front One generated word.

[0030] Please continue reading. Figure 1 As shown, in one embodiment of the present invention, the improved autoregressive generative model is compatible with the XGLM / OPT architecture, and the soft prompting implementation logic is consistent with GPT2, only needing to be adapted to the layer structure and embedding dimension of the corresponding decoder.

[0031] Please see Figure 1 The present invention provides an embodiment of an image caption generation system that combines scene graph retrieval with soft and hard prompts, including a memory and a processor capable of implementing the method steps. Specific Implementation S1. Acquire the input image, generate an image scene graph, and extract the core entities of the image. The image scene graph is defined as follows: ; It is a triplet relation; S2. Based on the global CLIP feature vector of the image, the candidate caption set is coarsely filtered by the FAISS index to obtain a coarsely filtered candidate caption subset; S3. The coarsely selected candidate caption subset is transformed into a language scene graph set. The image scene graph and the language scene graph are encoded by a shared graph convolutional network (GCN). Cross-modal semantic matching score and structured matching score are calculated respectively. The inverse ranking fusion (RRF) strategy is used to combine the scores and perform fine screening to output the top m candidate captions. S4. Based on the core entity of the image and the first m candidate subtitles, construct hard prompts and soft prompts respectively to form a soft and hard prompt collaborative input; S5. Input the soft and hard prompts together into the autoregressive generation model, and inject the core entity features into the deep network of the generation model through the deep injection mechanism to output image captions.

[0033] Furthermore, the method for extracting the core entity of the image in step S1 is as follows: Using the UniDet target detector from The confidence score (v) of the entity is obtained. First, the image entities are sorted in descending order by score (v). Then, the core entities are filtered through a two-layer deduplication strategy (exact match deduplication + word-level inclusion deduplication). ,satisfy: ={v1,v2,…,vk}, where k≤5, vi are the top k entities that pass the deduplication verification after sorting, and satisfy: 1) The entity clean_name must be unique; 2) The word set of an entity is not included in the word set of the selected entities (to avoid semantic redundancy); 3) If no valid entity is found after filtering, use the placeholder "(some objects)" as the core entity.

[0034] Furthermore, the coarse screening method described in step S2 is as follows: A) Image Feature Encoding: The CLIP model is used to globally encode the input image, resulting in a global feature vector. ∈ Where D is the CLIP feature dimension, and for Perform L2 normalization; B) Normalized Input a pre-built FAISS index and retrieve the Top-K1 candidate captions that are most similar to the image features, where K1=15; C) Anti-fraud filtering: Based on the "candidate caption-image ID" mapping relationship constructed by splitting the Karpathy dataset, candidate captions that are the same as the input image are filtered to obtain a coarsely filtered subset. .

[0035] Furthermore, the implementation method of scene graph encoding in step S3 is as follows: A) For the initial screening of candidate subtitles Constructing a language scene graph ,in A collection of text entities For syntactic relation triples between text entities; B) Node feature initialization: 1) Image scene graph node features: for The bounding box region corresponding to each entity in the image is cropped, and the region features are encoded using the CLIP model to obtain the image node features. , ; 2) Language scene graph node features: for The text name of each entity is encoded, and the text features are encoded using the CLIP model to obtain language node features. , ; C) Image scene graph encoding: 1) Construct an adjacency matrix with self-loops: ,in The original adjacency matrix, It is the identity matrix; 2) Symmetric normalization: Calculate the adjacency matrix degree matrix The normalized adjacency matrix is ​​obtained as follows: ; 3) Message Passing and Aggregation: Perform graph convolution message passing operations. ,in The image node feature matrix is ​​used; max pooling is performed on the aggregated node features to obtain the global embedding of the image scene graph. ; D) Language Scene Graph Encoding: Using a GCN network that is completely shared with the image scene graph, the same normalization, message passing, and pooling operations as in steps BC are performed to obtain the global embedding of the language scene graph. The specific formula is as follows: ; in The original adjacency matrix of the language scene graph. This is the degree matrix. The feature matrix of language nodes; E) Cross-modal semantic matching score: ; in Let cosine similarity function be defined as follows: ; F) Structured matching score: ; G) RRF Fusion Overall Score: in For smoothing parameters, They are respectively Rank after sorting in descending order; H) Fine-tuning: By The top-m candidate subtitles are filtered in descending order, where m=4, to obtain a finely filtered subset. .

[0036] Furthermore, the method for constructing the soft and hard prompts in step S4 is as follows: A) Hard hints build: ; in, These are the first m candidate subtitles output in step 3. For the core entity set of the image; B) Soft suggestion construction: 1) Core entity text encoding: A frozen CLIP model is used for... Text encoding is performed to obtain cross-modal entity features, using the following formula: ; in For the number of core entities, the CLIP model weights are frozen and do not participate in training; they are only used as feature extractors. 2) Cross-modal feature projection: Constructing a projection module The 512-dimensional CLIP entity features are mapped to the latent space of the decoder using the following formula: in For the decoder hidden layer dimension, This indicates a function composition operation, and the projection module is the only trainable soft cue-related component; 3) Hierarchical entity feature allocation: Based on the number of layers injected by the decoder... Evenly distributed, a hierarchical soft cue feature set (i.e., a deep injection feature set) is obtained: ; in For the number of injection layers, (B represents the batch size) The number of entities allocated to the i-th level), the allocation rule is as follows: If the number of entities is insufficient, it will be replenished cyclically; 4) Deep injection conditions: For the decoder... The layer performs soft hint (entity feature) injection when the following conditions are met: ; Where S=3 is the injection initiation layer. For injection layer spacing, For modulo operation, For hierarchical feature indexing; 5) Soft cue residual construction: Broadcastable residuals are constructed based on the hierarchical feature dimensions to achieve the fusion of soft cues and the decoder's hidden state. The formula is as follows: ; in To increase the sequence length dimension, To take the mean along the entity dimension, ensure that the residual dimension is compatible with the decoder's hidden state; 6) Soft hint deep injection execution: Combine the residual form of soft hints with the decoder's... Broadcasting addition in the hidden state of the layer enables soft hint injection, with the following formula: in For decoder number Hidden layer state, The sequence length; 7) Beam Search Batch Adaptation: When beam search is used during the inference phase, causing changes in batch size, the hierarchical soft cue features are expanded to match the batch size. The formula is as follows: ; in This is the original batch size. This is the current batch size. This is an expanded layered soft cue feature; C) Implementation of hardware and software prompts in tandem: 1) Hard cue embedding encoding: Convert hard cues into a token sequence and encode them into an embedding vector, using the following formula: ; in For hard hint token sequence, For hard-coded length, For the word embedding layer of the autoregressive model; 2) Attention Mask Correction: The attention mask is converted into a 4-dimensional tensor to adapt to the decoder's attention calculation. The formula is: ; 3) Collaborative Input Feature Construction: Soft cues (deep injection residuals), hard cue embeddings, and text prefix embeddings are fused together, using the following formula: ; in Embed the target subtitle prefix. For prefix length, For CLIP image global features, This is used to define soft cue residual features, enabling collaborative interaction between soft and hard cues and image features.

[0037] Furthermore, the method for generating image captions in step S5 is as follows: A) Generating model input construction: ; in For soft cue (entity projection feature) prefix embedding, For hard-signal word segmentation, This represents a sequence concatenation operation; B) Optimization objective: Employ an improved autoregressive generative model. The optimization objective is: ; in To generate a subtitle sequence, For sequence length, The core entity features after projection (soft hints); C) Word-by-word reasoning: The reasoning stage generates subtitles through word-by-word prediction, using the following formula: ; in For the t-th generated word, For the front One generated word.

[0038] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The above descriptions are merely preferred embodiments of the present invention and should not be construed as limiting the present application. All equivalent variations and modifications made within the scope of the claims of this invention should be considered within the scope of this invention.

Claims

1. A method for generating image captions based on scene graphs and entity-driven prompts, characterized in that, include: Step S1: Obtain the input image, generate an image scene graph, and extract the core entities of the image; Step S2: Based on the global CLIP feature vector of the image, the candidate caption set is coarsely filtered by the FAISS index to obtain a coarsely filtered candidate caption subset; Step S3: The coarsely screened candidate subtitle subset is transformed into a language scene graph set. The image scene graph and the language scene graph are encoded by the shared graph convolutional network GCN. The cross-modal semantic matching score and the structured matching score are calculated respectively. The inverse ranking fusion RRF strategy is used to combine the scores and perform fine screening to output the top m candidate subtitles. Step S4: Based on the core entity of the image and the first m candidate subtitles, construct hard cues and soft cues respectively to form a soft and hard cue collaborative input; Step S5: Input the soft and hard prompts together into the autoregressive generation model, and inject the core entity features into the deep network of the generation model through the deep injection mechanism to output image captions.

2. The image caption generation method based on scene graph and entity-driven prompts according to claim 1, characterized in that, The image scene graph in S1 is defined as follows: ； This is a triplet relation. This represents the original visual scene graph constructed from the input image. This represents the initial set of visual vertices containing physical entities in the image, identified by the object detector. This represents the set of directed relation triples defined on the vertex set to represent the topological interaction logic between objects.

3. The image caption generation method based on scene graph and entity-driven prompts according to claim 2, characterized in that, The method for extracting the core entities of the image in step S1 is as follows: Using the UniDet target detector from The confidence score (v) of the entity is obtained. First, the image entities are sorted in descending order by score (v), and then the core entities are filtered through a two-layer deduplication strategy. ,satisfy: ={v1,v2,…,vk}, where k≤5, vi are the top k entities that pass the deduplication verification after sorting, and satisfy: The names of entities after standardization processing, including conversion to lowercase, removal of punctuation marks, and removal of leading and trailing whitespace, are not duplicated; The word set of an entity is not included in the word set of the selected entities; If no valid entity is found after filtering, the placeholder "(some objects)" will be used as the core entity.

4. The image caption generation method based on scene graph and entity-driven prompts according to claim 2, characterized in that, Step S2 further comprises: Step A1, Image Feature Encoding: The CLIP model is used to globally encode the input image to obtain the global feature vector of the image. ∈ ,in, Let D represent the real number field space, where D is the CLIP feature dimension, and... Perform L2 normalization; Step A2: Normalize the Input the FAISS index, which is built offline based on an external large-scale image and text dataset, and retrieve the Top-K1 candidate captions that are most similar to the image features, where K1=15; Step A3, Anti-cheating Filtering: Based on the "candidate caption-image ID" mapping relationship constructed by splitting the Karpathy dataset, candidate captions that are the same image as the input image are filtered to obtain a coarsely filtered subset. .

5. The image caption generation method based on scene graph and entity-driven prompts according to claim 2, characterized in that, Step S3 further includes: Step B1: Coarsely screen candidate subtitles Constructing a language scene graph ,in A collection of text entities ； For syntactic relation triples between text entities; Step B2, Image scene graph node features: for For each entity in the image, the bounding box region corresponding to it is used to extract the region of interest (ROI) sub-image from the original image based on its bounding box coordinates. The CLIP model is then used to encode the region features to obtain the image node features. , ; Let D represent the real number field space, and let D represent the dimension of the feature vector. Language scene graph node features: (The rest of the text appears to be a list of features and their relationships.) The text name of each entity is encoded, and the text features are encoded using the CLIP model to obtain language node features. , ; Step B3: Construct an adjacency matrix with self-loops: ,in The original adjacency matrix, It is the identity matrix; Symmetric normalization: Calculating the adjacency matrix degree matrix The normalized adjacency matrix is ​​obtained as follows: ; Message Passing and Aggregation: Performing Graph Convolution Message Passing Operations ,in ； The image node feature matrix; Max pooling is performed on the aggregated node features to obtain the global embedding of the image scene graph: ； Step B4, Language Scene Graph Encoding: Using a GCN network that is completely shared with the image scene graph, perform the same normalization, message passing, and pooling operations as in steps B2-B3 to obtain the global embedding of the language scene graph. The specific formula is as follows: ； in Represents the identity matrix. This represents an adjacency matrix with added self-loop mechanism. The original adjacency matrix of the language scene graph. for The corresponding degree matrix, This represents the Laplace matrix after symmetric normalization. This is the initial language node feature matrix. This represents the hidden feature matrix of the language nodes after being updated by the graph convolutional layer. This indicates that maximum value filtering is performed at the node level. This represents the global embedding vector of the language scene graph obtained by aggregating through this max pooling operation; Step B5, Cross-modal semantic matching score: ； in This represents a subset of object-type entities selected from the text entity set. type(v')=object indicates that the semantic category attribute of entity v' is determined to be an object rather than an attribute or relation. Let cosine similarity function be defined as follows: ; Step B6, Structured Matching Score: ; Step B7, RRF fusion and comprehensive score: ； in For smoothing parameters, They are respectively Rank after sorting in descending order; Step B8, Detailed Screening: (Please click here) The top-m candidate subtitles are filtered in descending order, where m=4, to obtain a finely filtered subset. .

6. The image caption generation method based on scene graph and entity-driven prompts according to claim 2, characterized in that, Step S4 further comprises: Step C1, Hard hints construction: ； in, These are the first m candidate subtitles output in step 3. For the core entity set of the image; Step C2, Soft suggestion construction: Core entity text encoding: using a frozen CLIP model. Text encoding is performed to obtain cross-modal entity features, using the following formula: ； in For the number of core entities, the CLIP model weights are frozen and do not participate in training; they are only used as feature extractors. Cross-modal feature projection: Constructing a projection module The 512-dimensional CLIP entity features are mapped to the latent space of the decoder using the following formula: ； in For the hidden layer dimension of the decoder, This represents the layer normalization operation performed on the input features. This represents a fully connected layer that maps the feature dimension from 512 to H. Represents the activation function of the Gaussian error linear unit. This represents a fully connected layer whose dimensions remain unchanged. This represents the final layer normalization operation performed on the output features. This indicates the function composition operation. This indicates embedding the input entity. The projection module is the only trainable soft cue-related component, which maps the connection module to the decoder feature space. Hierarchical entity feature allocation: based on the number of decoder injection layers Evenly distributed, a hierarchical soft cue feature set is obtained: ； in For the number of injection layers, (B represents the batch size) The number of entities allocated to the i-th level), the allocation rule is as follows: If the number of entities is insufficient, it will be replenished cyclically; Deep injection conditions: For the decoder's first... A layer will perform soft hint injection if the following conditions are met: ； Where S=3 is the injection initiation layer. For injection layer spacing, For modulo operation, For hierarchical feature indexing; Soft cue residual construction: Broadcastable residuals are constructed based on the hierarchical feature dimensions, and soft cues are fused with the decoder's hidden state. The formula is as follows: ； in This represents the feature retrieved from the hierarchical feature stack and assigned to the current injection layer. Specific feature blocks; To increase the sequence length dimension, This is an operation to take the mean along the entity dimension, where This means that the number of dimensions of the tensor remains unchanged when calculating the mean (to prevent dimensional collapse) in order to support subsequent broadcast addition operations with the hidden state; Soft hint deep injection execution: The residual form of the soft hint is combined with the decoder's... Broadcasting addition in the hidden state of the layer enables soft hint injection, with the following formula: ； in For the decoder Hidden layer state, The sequence length; Beam Search Batch Adaptation: When beam search is used during the inference phase, resulting in changes in batch size, the hierarchical soft cue features are expanded to match the batch size. The formula is as follows: ； in This is the original batch size. This is the current batch size. This is an expanded layered soft cue feature; Step C3, Implementation of hardware and software prompts in tandem: Hard cue embedding encoding: Convert hard cues into a token sequence and encode them into an embedding vector, using the following formula: ； in For hard hint token sequence, For hard-coded length, For the word embedding layer of the autoregressive model; Attention mask correction: The attention mask is converted into a 4-dimensional tensor to fit the decoder's attention calculation. The formula is as follows: ； in For the original attention mask tensor; Indicates the number of dimensions of the tensor obtained; This means that when the input is 2-dimensional, two new dimensions of size 1 are inserted in the middle through slicing operations to expand it into a 4-dimensional tensor; This indicates that when the input is 3D, in The operation of inserting a new dimension of size 1 at index 2 of a tensor; Collaborative input feature construction: This involves fusing soft prompts, hard prompts, and text prefix embeddings, using the following formula: ； Where [;] represents a tensor concatenation operation along the sequence length dimension; This represents the concatenated complete input sequence embedding; This indicates that the feature matrix belongs to the real number space, and its dimension is (hard hint: length). +prefix length Decoder hidden layer dimension ; (·) denotes a learnable linear projection layer; This represents the image features after projection adaptation; This indicates that the feature matrix belongs to the real number space, and its dimension is equal to the length of the image feature sequence. x Decoder Hidden Layer Dimension ; (Q, K, V) represents a multi-head attention mechanism used to compute cross-modal interactions between the text query vector Q and the image key vectors K and V; This represents the output features of the cross-modal attention layer; This represents the fused features that are finally input to the decoder; the "+" in the formula represents the residual connection operation, which is to add the feature vectors from different sources element by element. Embed the target subtitle prefix. For prefix length, For CLIP image global features, To provide soft cue residual features, we perform collaborative interaction between soft and hard cues and image features.

7. The image caption generation method based on scene graph and entity-driven prompts according to claim 2, characterized in that, Step S5 further comprises: Step D1: Generate model input construction: ； in This refers to the operation of constructing a soft cue prefix sequence from the projected entity features. To represent converting a text string into a discrete representation that the model can recognize. Sequence segmentation operations, This represents the discrete hard hint instruction template used for bootstrapping. This represents the concatenation operation of tensors along the sequence dimension; Step D2, Optimization Objective: Employ an improved autoregressive generative model. The optimization objective is: ； in To generate a subtitle sequence, This represents the operation of finding the parameter variable that maximizes the probability function value (i.e., the optimal sequence). This represents the complete input sequence of the generated model, including soft prompt prefixes and hard prompt instructions, after concatenation. For sequence length, The core entity features after projection; Step D3, Word-by-Word Reasoning: The reasoning stage generates subtitles through word-by-word prediction. The formula is: ； in For the t-th generated word, For the front One generated word.

8. The image caption generation method based on scene graph and entity-driven prompts according to claim 7, characterized in that, The improved autoregressive generative model is compatible with the XGLM / OPT architecture, and the soft prompting implementation logic is consistent with GPT2. It only needs to be adapted to the layer structure and embedding dimension of the corresponding decoder.

9. A scene graph retrieval and hardware / software cues collaborative image caption generation system, characterized in that, Includes a memory and a processor, and is capable of implementing the image caption generation method based on scene graph and entity-driven prompts according to any one of claims 1-7.

Images