Dense image description generation method, system, device, medium and program product

Abstract

The application discloses a dense image description generation method, system, device, medium and program product, relates to the technical field of dense description, and comprises the following steps: extracting a region of interest in a to-be-processed image, extracting proposal features of each region of interest and adjacent features of each adjacent region of interest of the region of interest, determining a semantic context relationship in this way, determining a spatial context relationship according to a center position of each region of interest and center positions of each adjacent region of interest, and combining the semantic context relationship and the spatial context relationship to obtain a visual embedding; extracting a visual object in the to-be-processed image, searching for a target triple related to the visual object from a knowledge graph, and encoding the target triple into a knowledge embedding; after the visual embedding and the knowledge embedding are mapped to the same dimension, the visual embedding and the knowledge embedding are mutually embedded with features, a semantic description sentence is obtained through decoding, and the performance of dense description generation is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of dense description, and particularly to a dense image description generation method, system, device, medium and program product. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] Dense description technology covers object detection, semantic content analysis and natural language description. With the release of large-scale description data sets and the progress of deep learning technology, the performance of dense description has been significantly improved, and its application is widespread, including human-computer interaction, image-text matching, visual question answering, etc. Dense description is more advanced than image description, providing richer and more detailed description of images. A fully convolutional localization network (FCLN) is proposed in the prior art, which includes a detector to identify all regions of interest (RoI) and generate their text descriptions.

[0004] In existing research, it is roughly divided into two stages according to whether context information is included. The early architecture uses a faster region convolutional network (Faster R-CNN) to detect RoI and uses a long short-term memory network (LSTM) to generate text description; however, these frameworks mainly focus on RoI, ignoring potentially valuable context information. Subsequent research addresses this issue by integrating RoI features with global image features before generating text descriptions, but the global context is too rough. Some methods also study fine-grained context information, such as a non-local similarity map is proposed to facilitate feature interaction between target RoI and its neighboring RoIs, and a data-driven method is used to reveal the close relationship between RoI and detected objects, thus an architecture considering context information is proposed, but it still fails to effectively utilize the potential advantages of context clues. Language clues from surrounding RoIs are always valuable, and when generating descriptions for a specific RoI, descriptions obtained by focusing only on the RoI without combining context information to observe the surrounding environment are not comprehensive or rich.

[0005] Secondly, a Transformer-based dense descriptor (TDC) is proposed based on the Transformer architecture, which assigns different importance to each detected RoI through a region-object correlation score unit (ROCSU). However, existing context modeling mechanisms heavily rely on the quality of object proposals. Due to the presence of blurred images or small targets, low-quality object proposals are inevitably generated, which contain less semantic information.

[0006] In addition, the existing dense description method inadvertently involves unbalanced learning in the learning process, mainly due to uneven distribution of training data. The trained model tends to generate descriptions containing frequently occurring words in the training samples, resulting in limited vocabulary diversity and insufficient object attribute description. To solve this problem, some methods propose a dynamic vocabulary frequency histogram resampling strategy to rebalance words of different frequencies during training. However, this method often falls short in understanding the inherent reasons for certain visual attributes, such as the common color green for vegetables, which goes beyond visual features and covers higher-level common sense reasoning, and existing methods cannot make accurate explanations. SUMMARY

[0007] To solve the above problems, the present application provides a dense image description generation method, system, device, medium and program product, which generates dense image description based on external knowledge graph and visual context perception, ensures the consistency of visual local description and context, and effectively improves the performance of dense description generation by capturing the attributes of visual objects and their topological relationships with other objects.

[0008] To achieve the above purpose, the present application adopts the following technical solutions:

[0009] In a first aspect, the present application provides a dense image description generation method, comprising:

[0010] obtaining an image to be processed;

[0011] extracting a region of interest in the image to be processed, extracting proposal features of each region of interest and adjacent features of each region of interest adjacent to it to determine the semantic context relationship of each region of interest, determining the spatial context relationship of each region of interest according to the center position of each region of interest and the center position of each region of interest adjacent to it, and combining the semantic context relationship and the spatial context relationship to obtain visual embedding;

[0012] extracting visual objects in the image to be processed, retrieving target triples related to the visual objects from the knowledge graph, and encoding the target triples into knowledge embedding;

[0013] After mapping the visual embedding and the knowledge embedding to the same dimension, the visual embedding and the knowledge embedding are mutually embedded, and the corresponding semantic description sentence is obtained by decoding after embedding.

[0014] As an optional implementation, the process of determining the semantic context relationship of each region of interest comprises: splicing the adjacent features of each adjacent region of interest and the proposal features of the region of interest, and determining the semantic context relationship of each region of interest through a weight matrix W p o represents the rectangular frame of the i-th adjacent region of interesti and the rectangle of the region of interest o p The semantic context relationship between them is determined, and the learned weight matrix W is learned using the Tanh activation function. p All elements are constrained to the range (-1, 1).

[0015] As an alternative implementation, the process of determining the spatial context of each region of interest includes: determining the center position c of each adjacent region of interest. i and the center position of the region of interest c p The data is concatenated using the weight matrix W. q The rectangle o represents the i-th adjacent region of interest. i and the rectangle of the region of interest o p The spatial context relationship between them, and the learned weight matrix W is obtained by using the ReLU activation function. q All elements are constrained to the range [0, +∞).

[0016] As an alternative implementation method, the process of extracting knowledge embedding includes:

[0017]

[0018] Among them, t + The target triple is f; f is a nonparametric scoring function used to calculate the input text representation E through the dot product. θ (x) and triplet represent E θ Similarity between (t); E θ (t) indicates the use of sentence encoder E θ The process of representing a triple t is as follows: the triple is represented as a sequence of tags, consisting of entity and relation tags and delimiters between them. Then, the triple tags are passed to the sentence encoder to obtain the triple representation; G is a set of fact triples; t is a triple defined for a specific fact.

[0019] Finally, the retrieved target triples are embedded and encoded into knowledge embeddings using a word segmenter.

[0020] As an alternative implementation, the process of mutual feature embedding between visual embedding and knowledge embedding includes:

[0021] When passing features from knowledge embedding to visual embedding, the knowledge embedding is used as the key K and value V, and the visual embedding is used as the query Q; when passing features from visual embedding to knowledge embedding, the knowledge embedding is used as the query Q, and the visual embedding is used as the key K and value V.

[0022] As an alternative implementation, after the feature embedding, the obtained enhanced visual embedding and enhanced knowledge embedding are restored to the original dimension and spliced with the original embedding through a residual connection; the enhanced visual embedding after the residual connection is projected to the language feature space through the alignment network, and then input into the text decoder together with the enhanced knowledge embedding after the residual connection, to obtain the corresponding semantic description sentence.

[0023] In a second aspect, the present application provides a dense image description generation system, comprising:

[0024] The acquisition module is configured to acquire a to-be-processed image.

[0025] The visual embedding extraction module is configured to extract a region of interest in the to-be-processed image, extract a proposal feature of each region of interest and a neighboring feature of each region of interest adjacent to the region of interest, determine a semantic context relationship of each region of interest in this way, determine a spatial context relationship of each region of interest according to a center position of each region of interest and a center position of each region of interest adjacent to the region of interest, and combine the semantic context relationship and the spatial context relationship to obtain a visual embedding.

[0026] The knowledge embedding extraction module is configured to extract a visual object in the to-be-processed image, retrieve a target triple related to the visual object from a knowledge graph, and encode the target triple into a knowledge embedding in this way.

[0027] The alignment decoding module is configured to map the visual embedding and the knowledge embedding to the same dimension after the mapping, perform feature embedding on the visual embedding and the knowledge embedding, and obtain a corresponding semantic description sentence through decoding after the feature embedding.

[0028] In a third aspect, the present application provides an electronic device, comprising a memory and a processor, and computer instructions stored in the memory and running on the processor, wherein when the computer instructions are run by the processor, the method of the first aspect is completed.

[0029] In a fourth aspect, the present application provides a computer readable storage medium for storing computer instructions, wherein when the computer instructions are executed by a processor, the method of the first aspect is completed.

[0030] In a fifth aspect, the present application provides a computer program product comprising a computer program, wherein when the computer program is executed by a processor, the method of the first aspect is completed.

[0031] Compared with the prior art, the present application has the following beneficial effects:

[0032] The application provides a dense image description generation method based on an external knowledge graph and visual context perception, ensures consistency of visual local description and context, simultaneously captures attributes of visual objects and topological relations of the visual objects with other objects, and effectively improves performance of dense description generation.

[0033] The application provides a novel common sense knowledge graph CSKG and a direct knowledge retrieval framework. The method provides prior guidance for image-to-text conversion, solves a semantic level answer retrieval problem by only using representation similarity between visual objects and triples, connects visual objects and named entities through a multi-modal knowledge graph, and simultaneously captures relations between the entities by using obtained external knowledge.

[0034] The application provides a context extractor CE, which promotes sharing and message passing of spatial and semantic context information, and related visual object features are passed to a frozen visual Transformer (ViT) to obtain visual embeddings.

[0035] The application introduces a dual-stream interaction module DSIM, enhances interaction between visual embeddings and text knowledge embeddings, and thus improves accuracy of region-level description.

[0036] Advantages of additional aspects of the application will be partially given in the following description, partially become obvious from the following description, or be learned by practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description only constitute the embodiments of the application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of the provided drawings.

[0038] Figure 1 A dense image description generation method flowchart is provided for the embodiment 1 of the application;

[0039] Figure 2 A visual embedding extraction module schematic diagram is provided for the embodiment 1 of the application;

[0040] Figure 3 A knowledge embedding extraction module schematic diagram is provided for the embodiment 1 of the application;

[0041] Figure 4 An alignment decoding module schematic diagram is provided for the embodiment 1 of the application;

[0042] Figure 5 An experimental result schematic diagram is provided for the embodiment 1 of the application. DETAILED DESCRIPTION

[0043] The present application will be further described with reference to the drawings and examples.

[0044] It should be noted that the following detailed description is exemplary in nature and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0045] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments in accordance with the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0046] The embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0047] Term explanation:

[0048] Image caption generation: Image caption generation is a technique in computer vision and natural language processing that aims to generate a natural language description of an image as a whole. This technique typically uses convolutional neural networks (CNNs) in deep learning to extract image features and uses recursive neural networks (RNNs) or Transformer models to generate descriptions.

[0049] Dense image caption generation: Dense image caption generation is an extension of image caption generation, aiming to generate detailed descriptions for multiple regions in an image, rather than just a holistic description of the entire image. This technique can capture finer details and multiple elements in an image, usually by combining object detection and description generation techniques.

[0050] Visual context awareness: Visual context awareness refers to computer vision systems understanding and utilizing contextual information in images or videos for more accurate recognition and analysis. For example, when identifying an object in a scene, not only the independent features of the object are considered, but also the information of its surrounding environment and other related objects. This context awareness can improve the accuracy and robustness of the model, making it more effective in real-world applications.

[0051] Knowledge Retrieval: Knowledge retrieval is an information retrieval technique aimed at extracting relevant information or answers from vast knowledge bases or databases. It often involves natural language processing techniques to understand user queries and match and retrieve the most relevant knowledge snippets.

[0052] Multimodal Learning: Multimodal learning is a machine learning approach that combines and utilizes data from multiple modalities (such as text, images, audio, video, etc.) for training and inference. The core of this approach is to improve the model's understanding and prediction capabilities by fusing different types of data sources.

[0053] Embodiment 1

[0054] This embodiment proposes a dense image description generation method, which includes:

[0055] Obtaining a to-be-processed image;

[0056] Extracting a region of interest in the to-be-processed image, extracting a proposal feature for each region of interest and a neighboring feature of each region of interest adjacent to it, determining a semantic context relationship of each region of interest, determining a spatial context relationship of each region of interest according to a center position of each region of interest and a center position of each region of interest adjacent to it, and combining the semantic context relationship and the spatial context relationship to obtain a visual embedding;

[0057] Extracting a visual object in the to-be-processed image, retrieving a target triple related to the visual object from a knowledge graph, and encoding the target triple into a knowledge embedding;

[0058] After mapping the visual embedding and the knowledge embedding to the same dimension, the visual embedding and the knowledge embedding are mutually embedded with features, and the corresponding semantic description sentence is obtained through decoding after the feature embedding.

[0059] As Figure 1 shown, this embodiment proposes a dense image description network based on external knowledge graph and visual context perception (EKCA-Cap), which includes a visual embedding extraction module, a knowledge embedding extraction module, and an alignment decoding module. In addition, multiple pre-trained frozen models are used to extract features, and the multiple pre-trained frozen models include a frozen Faster R-CNN model, a frozen ViT model, a frozen DE (Detection Everything) model, a frozen tokenizer embedding, an alignment network, and a text decoder.

[0060] Specifically:

[0061] In the visual embedding extraction module, first, a CNN model is used to learn low-level semantic features from the input image to be processed in the manner of a Faster R-CNN model; then, a set of regions of interest is generated from the semantic features through a region proposal network (RPN); then, a context feature extractor is introduced to generate visual features related to each region of interest RoI, which are input into a ViT model to extract rich and fine-grained visual embeddings.

[0062] In the knowledge embedding extraction module, first, a DE model is used to identify all visual objects in the input image to be processed; then, knowledge related to the visual objects is retrieved from a common-sense knowledge graph, and these knowledge words are encoded into knowledge embeddings.

[0063] In the alignment decoding module, the generated visual embeddings and knowledge embeddings exchange information through a double-flow interaction module to solve the misalignment problem generated by each module; then the visual embeddings are projected into the language feature space through an alignment network and input into the text decoder together with the knowledge embeddings to generate dense descriptions.

[0064] The overall framework of the EKCA-Cap network will be described in detail below.

[0065] 1. Visual embedding extraction module.

[0066] Define the region of interest in the image to be processed as M = {M i |i = 1, 2,..., N}, the global region (i.e. the entire image) as M * , and the adjacent region of interest as M in = M i / M in , where M i is the nth adjacent region of interest of the ith region of interest. For each region of interest M ip , its proposal feature, adjacent feature and global feature are represented as V in , V ig and V in , respectively.

[0067] where the adjacent feature V in is V i = f(M in , M p ), where f(·) represents a visual context feature extraction module that extracts features through spatial context and visual context, respectively.

[0068] The proposal feature V p is obtained through a single-layer convolutional network. Specifically:

[0069] The global region is input into CenterNet to extract keypoints; CenterNet is an anchorless detector. This step is slightly modified to improve recall. Since contextual information is merged in the low-resolution feature map, it is easier to coarsely locate objects. Following the CenterNet workflow, keypoint features c with different confidence scores are obtained. i and visual features c p Since the confidence proposal is related to neighboring objects that contain rich contextual information, these neighboring objects are used to assist in proposing features v. p Description generation.

[0070] In this embodiment, the visual context is modeled by constructing a relationship graph G = (V, E, g); where v i ∈V represents the neighboring features of adjacent regions of interest, and g represents the global region M. * eigenvectors, e ip ∈E represents v i and v p The relationship between them.

[0071] In this embodiment, e ip This represents the relationship between a scene and an object, and its constant value is set to 1; specifically:

[0072] e ip =ReLU(W q [c i ,c p ])*Tanh(W p [v i ,v p ]);

[0073] Among them, c i and c p The rectangle o represents the i-th adjacent region of interest. i and the rectangle of the region of interest o p The central location, W q and W p It is a learnable weight matrix.

[0074] like Figure 2 As shown, the final context relationship is obtained by merging semantic context relationships and spatial context relationships; specifically:

[0075] (1) Learning semantic context relations; First, the neighboring features v of each adjacent region of interest are... i and the proposed features of the region of interest v p Perform concatenation; then, through the weight matrix W p , to represent o i and o psemantic context relationship between o p and o i ; in addition, all elements in the learned weight matrix W p are constrained in the range of (-1, 1) using the activation function Tanh.

[0076] (2) Learning of spatial context relationship; first, the center positions c i and c p of each adjacent region of interest are spliced; then, the spatial context relationship between o q and o i is represented by the weight matrix W p ; since the value of the spatial context relationship is non-negative, all elements in the learned weight matrix W q are constrained in the range of [0, +∞) using the activation function ReLU.

[0077] (3) Multiplying the semantic context relationship and the spatial context relationship to obtain the scene object feature e ip , and inputting the scene object feature e ip into the ViT model to extract the final visual embedding F v .

[0078] 2. Knowledge embedding extraction module.

[0079] Let E be a set of entities and R be a set of relationships. A specific fact is defined as a triple: t = (e h , r, e t ) ∈ E × R × E, where e h and e t are the head entity and the tail entity, respectively, and r is the relationship between them. In addition, a knowledge graph is defined as a set of fact triples:

[0080] It is worth noting that this knowledge graph is a valuable source of knowledge for many natural language applications. However, the input of most natural language tasks is usually represented as a token sequence: x = [w1, w2,..., w |x| ].

[0081] Suppose that given input x, t + is a target triple that needs to be retrieved. The goal of the traditional fact retrieval process is usually formalized as the following three consecutive tasks:

[0082]

[0083] where, is a mention detection model, m is an entity mention detected in the input x, p φ (e|m, x) is an entity disambiguation model, p θ(t, e, x, G) is a relation classification model.

[0084] However, this three-step method has some limitations. First, they are prone to accumulate errors. Second, due to its decomposition structure, it is difficult to train the three sub-modules in an end-to-end manner, and labels need to be provided for the training of each sub-module.

[0085] To solve the above challenges of existing knowledge graph fact retrieval methods, the present embodiment proposes a direct knowledge retrieval framework as shown in Figure 3 The goal of the present embodiment is to use a single sentence encoder model E θ to simplify the process and introduce no additional variables (e.g., m and e) as follows:

[0086]

[0087] where f is a non-parametric scoring function used to calculate the similarity between the input text representation E θ (x) and the triple representation E θ (t) by dot product; E θ (t) represents the process of representing the triple t using the sentence encoder E θ , specifically: first, represent the triple t as a token sequence: t = [w1, w2,..., w |t| ], composed of entity and relation tokens and a separator token (i.e., special token [SEP]) between them; then pass the triple token to E θ to obtain the triple representation; finally, encode the target triple representation into a knowledge embedding through a tokenizer.

[0088] 3. Alignment decoding module.

[0089] Since the various common sense elements contained in the knowledge graph derived from the common sense knowledge graph may not be directly related to the current visual context, this brings the challenge of aligning multiple knowledge embeddings with a single visual embedding, which is referred to as the "variation problem" of common sense knowledge. To solve this challenge, the present embodiment introduces a dual-flow interaction module, which aims to enhance the exchange of information between visual embeddings and knowledge embeddings, thereby improving the alignment effect.

[0090] As shown in Figure 4 , the dual-flow interaction module includes an adaptation layer and a cross-modal interaction module.

[0091] Specifically:

[0092] The adaptation layer consists of a single linear layer, which is used to adapt the visual embedding F v or the knowledge embedding F kmapping to a low-dimensional latent space, or mapping them back to the original feature space. In the dual-stream interaction module, first, the visual embedding and the knowledge embedding are mapped to the same latent space through an adaptation layer.

[0093] Second, the features in the knowledge embedding are transferred into the visual embedding through the cross-modal interaction layer and the feed-forward layer, where the knowledge embedding is taken as the key K and the value V, and the visual embedding is taken as the query Q; meanwhile, the features in the visual embedding are transferred into the knowledge embedding through the cross-modal interaction layer and the feed-forward layer, where the knowledge embedding is taken as the query Q, and the visual embedding is taken as the key K and the value V.

[0094] Then, after the feature embedding, the obtained enhanced visual embedding and enhanced knowledge embedding are restored to their original feature space and spliced with the original embedding through a residual connection;

[0095] Finally, the enhanced visual embedding after the residual connection is projected into the language feature space through an alignment network, and then input into the text decoder together with the enhanced knowledge embedding after the residual connection, to obtain the corresponding semantic description sentence. The alignment network adopts this structure to achieve more effective embedding alignment.

[0096] 4. Experiments and results.

[0097] The Visual Genome (VG), VG-COCO and RefCOCOg datasets are used as evaluation benchmarks, and METEOR and CIDEr are used as evaluation indicators. These experiments are carried out using four NVIDIA 3090 graphics processors running Ubuntu 18.04.

[0098] (1) mAP index comparison: the method EKCA-Cap of the embodiment is compared with other baseline methods, as shown in Table 1. The results show that the EKCA-Cap reaches 15.82% mAP on the VG V1.0 dataset, 15.65% on the VG V1.2 dataset, and 18.20% on the VG-COCO dataset, which is significantly better than the existing state-of-the-art method (SOTA).

[0099] Specifically, compared with the SOTA method DCMSTRD, the method of the embodiment obtains a 2.21 mAP gain on the VG V1.2 dataset. These performance improvements are mainly attributed to the advantages of the context feature extractor and the integration of text knowledge in the common sense knowledge graph with visual objects, which enhances the refinement of the captions, including object property and topological relationship caption generation.

[0100] To further demonstrate the capability of the EKCA-Cap model of the embodiment in generating dense captions, the influence of visual localization is ignored. In the case where the ground truth bounding boxes are provided, the mAP of EKCA-Cap on VG V1.0, VG V1.2 and VG-COCO reaches 46.65%, 44.32% and 49.35% respectively, which is 12.45% higher than that of BLIP2 on VG-COCO.

[0101] Table 1. mAP of models on VG and VG-COCO datasets

[0102]

[0103]

[0104] (2) Comparison of Meteor and Cider indicators: The comparative experiment uses the same settings as the previous study, as shown in Table 2. On the RefCOCOg dataset, EKCA-Cap significantly outperforms these methods, with METEOR scores increasing from 16.6 to 17.2 and CIDEr scores increasing from 108.3 to 113.4, achieving significant improvements compared to the previous SOTA method Osprey. This improvement is mainly due to the CE module in EKCA-Cap, which provides a broader context through the text context of other neighboring RoIs during caption generation. In addition, CSKG is able to balance the frequency of word usage, resulting in the above excellent results.

[0105] The embodiment further evaluates the EKCA-Cap method on the VG V1.2 dataset. The METEOR score results are shown in the fourth column of Table 2, and the CIDEr score results are shown in the fifth column. Obviously, the EKCA-Cap method proposed in the embodiment has a relative improvement of 2.3 in METEOR score and 23.1 in CIDEr score compared to the Alpha-CLIP+LLaVA method. It is worth noting that the performance gap between the method proposed in the embodiment and the SOTA method on the VG V1.2 dataset is smaller than that on the RefCOCOg dataset. This difference may be due to the larger size of the VG V1.2 dataset, which contains more images with complex scenes and captions, making it more challenging to generate captions even with the guidance of CSKG and CE.

[0106] Table 2. METEOR and CIDEr indicators of models on VG and RefCOCOg datasets

[0107]

[0108]

[0109] (3) The effectiveness of the components CSKG, CE and DSIM proposed in this embodiment is demonstrated by ablation experiments. DSIM aims to maximize the interaction between visual and textual knowledge modalities. Therefore, when CSKG is removed, DSIM should also be omitted. By introducing CSKG and DSIM, the mAP on the VG V1.2 dataset is improved by 2.06% (from 9.52% to 11.58%) (Table 3, rows 1-2). Similarly, the introduction of the context extractor (CE) improves the mAP on the VG V1.2 dataset by 4.8% (from 9.52% to 14.32%) (Table 3, rows 1-3). On the VG-COCO dataset, the introduction of CSKG and DSIM brings a 3.05% improvement, while the introduction of CE brings a 4.94% improvement. These significant improvements highlight the importance of CSKG, CE and DSIM. In particular, the CE module integrates multi-scale context and contextual cues between individual regions in region generation. Contextual cues, including global and neighboring information, make significant contributions to improving the final performance.

[0110] Table 3 Ablation experiments

[0111]

[0112] The following demonstrates the qualitative results and analysis of the method of this embodiment to provide a subjective evaluation of the method of this embodiment. Figure 5 Examples of four dense caption results are shown in FIG. 8. All RoIs and their descriptions are visualized. Specifically, Figure 5 (a) and (c) in FIG. 8 show examples in the VG-COCO dataset, Figure 5 (b) and (d) in FIG. 8 show examples in the VG V1.2 dataset. As Figure 5 As shown in (a) and (b) in FIG. 8, the EKCA-Cap proposed in this embodiment performs high quality in RoI localization and caption generation. Notably, the model of this embodiment demonstrates proficient grammar usage, and most of the generated sentences are readable and conform to correct English grammar and human understanding standards. In addition, the region captions include attribute information of visual targets, such as “gree grass”, “white shirt” and “blue jeans” in (a), and “brown wooden floor” in (b), which is mainly attributed to CSKG, which provides information about common object attributes and their relationships.

[0113] The EKCA-Cap method proposed in this embodiment can effectively detect RoIs of different scales. For example, in Figure 5In (a), it accurately identifies and describes the RoI containing two people standing on the grass, each with different scales. In (b), the method of the present embodiment identifies many objects related to the kitchen, such as the dishwasher and microwave, and the generated region captions are mostly related to the kitchen. This capability is attributed to the guidance of CE, which provides linguistic contextual cues from surrounding RoIs during caption generation.

[0114] The present embodiment also demonstrates the dense caption results of the baseline model COCG, as shown in (c) and (d). The method proposed in the present embodiment is capable of identifying small objects, such as the red frisbee clearly identified in the first example. This capability is attributed to the guidance of CSKG, which provides common sense understanding. It is evident from the context that a man is bending down and handing something to another person on the grass. In the second example shown in (b), the CSKG and CE proposed in the present embodiment are capable of extracting rich information from the image. Specifically, according to the context, it can be judged that the environment is a kitchen. Therefore, captions such as "a black dishwasher", "a black microwave" and "a clear plastic bottle" are generated, covering objects and their attributes related to the kitchen. For the baseline model COCG, its results lack relevant descriptions of the kitchen, as shown in (d). Figure 5 Figure 5

[0115] Embodiment 2

[0116] The present embodiment provides a dense image description generation system, comprising:

[0117] An acquisition module configured to acquire a to-be-processed image;

[0118] A visual embedding extraction module configured to extract a region of interest in the to-be-processed image, extract proposal features of each region of interest and adjacent features of each region of interest adjacent thereto to determine semantic contextual relationships of each region of interest, determine spatial contextual relationships of each region of interest according to a center position of each region of interest and center positions of each region of interest adjacent thereto, and combine the semantic contextual relationships and the spatial contextual relationships to obtain visual embeddings;

[0119] A knowledge embedding extraction module configured to extract visual objects in the to-be-processed image, retrieve target triples related to the visual objects from a knowledge graph, and encode the target triples into knowledge embeddings;

[0120] An alignment decoding module configured to map the visual embeddings and the knowledge embeddings to the same dimension, and then perform feature embedding on the visual embeddings and the knowledge embeddings, and obtain corresponding semantic description sentences through decoding after the feature embedding.​​

[0121] In this embodiment, the process of determining the semantic context relationship of each region of interest includes splicing the adjacent features of each adjacent region of interest and the proposed features of the region of interest, representing the semantic context relationship between the i-th adjacent region of interest and the region of interest by a weight matrix W p i p and the center position c p of the region of interest using an activation function Tanh to constrain all elements in the learned weight matrix W p to be within the range of (-1, 1).

[0122] In this embodiment, the process of determining the spatial context relationship of each region of interest includes splicing the center positions c i of each adjacent region of interest and the region of interest, representing the spatial context relationship between the i-th adjacent region of interest and the region of interest by a weight matrix W q i p and the center position c p of the region of interest using an activation function ReLU to constrain all elements in the learned weight matrix W q to be within the range of [0, +∞).

[0123] In this embodiment, the process of extracting knowledge embedding includes:

[0124]

[0125] where t + is a target triple; f is a non-parametric scoring function for calculating the similarity between the input text representation E θ (x) and the triple representation E θ (t) by dot product; E θ (t) represents the process of representing a triple t using a sentence encoder E θ , specifically: representing the triple as a token sequence composed of entity and relation tokens and a separation token between them, and then passing the triple token to the sentence encoder to obtain the triple representation; G is a set of factual triples; t is a triple defined by a specific fact;

[0126] Finally, the retrieved target triple is encoded into knowledge embedding by a tokenizer.

[0127] In this embodiment, the process of feature embedding of the visual embedding and the knowledge embedding includes:

[0128] ​​When transferring the features in the knowledge embedding into the visual embedding, taking the knowledge embedding as the key K and the value V, and taking the visual embedding as the query Q; when transferring the features in the visual embedding into the knowledge embedding, taking the knowledge embedding as the query Q, and taking the visual embedding as the key K and the value V.

[0129] In this embodiment, after the feature embedding, the obtained enhanced visual embedding and enhanced knowledge embedding are restored to the original dimension, and are spliced with the original embedding through a residual connection; the enhanced visual embedding after the residual connection is projected into the language feature space through an alignment network, and then is input into the text decoder together with the enhanced knowledge embedding after the residual connection, to obtain the corresponding semantic description sentence.

[0130] It should be noted that the above modules correspond to the steps described in Embodiment 1, and the above modules have the same examples and application scenarios as the corresponding steps, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules as part of the system can be executed in a computer system such as a set of computer executable instructions.

[0131] In more embodiments, there are also provided:

[0132] An electronic device includes a memory and a processor, and computer instructions stored in the memory and running on the processor, when the computer instructions are run by the processor, the method described in Embodiment 1 is completed. For brevity, it will not be repeated here.

[0133] It should be understood that in this embodiment, the processor can be a central processing unit CPU, and the processor can also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, ready-to-program gate arrays FPGA or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0134] The memory can include read-only memory and random access memory, and provide instructions and data to the processor, and a part of the memory can also include non-volatile random access memory. For example, the memory can also store device type information.

[0135] A computer readable storage medium for storing computer instructions, when the computer instructions are executed by the processor, the method described in Embodiment 1 is completed.

[0136] The method in embodiment 1 can be directly embodied as being completed by a hardware processor or being completed by a combination of hardware and software modules in the processor. The software modules can be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, or the like. The storage medium is located in a memory, and a processor reads information in the memory and completes the steps of the above method in combination with hardware. To avoid repetition, no longer detailed description is made herein.

[0137] A computer program product comprises a computer program, which, when executed by a processor, implements the method described in embodiment 1.

[0138] The present application also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions embodied in program modules, executed by devices in the target real or virtual processor to perform processes / methods as described above. Generally, program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform particular tasks or implement particular abstract data types. In various embodiments, the functionality of program modules can be combined or split between program modules as desired. Machine-executable instructions for program modules can be executed within a local or distributed device. In a distributed device, program modules can be located in local and remote storage media.

[0139] Computer program code for carrying out operations of the present application can be written in one or more programming languages. These computer program code can be provided to a processor of a general purpose computer, a special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the computer or other programmable data processing apparatus, causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be entirely on the computer, partially on the computer, as a stand-alone software package, partially on the computer and partially on a remote computer, or entirely on a remote computer or server.

[0140] In the context of the present application, computer program code or related data can be carried by any suitable carrier to enable a device, apparatus or processor to perform various processes and operations described above. Examples of carriers include signals, computer-readable media, and the like. Examples of signals can include electrical, optical, radio, sound or other forms of propagated signals, such as carrier waves, infrared signals, and the like.

[0141] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the present embodiment can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software manner depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0142] Although the specific embodiments of the present application are described above in combination with the drawings, it is not a limitation on the scope of protection of the present application. Those skilled in the art should understand that various modifications or variations made by those skilled in the art on the basis of the technical solutions of the present application without creative labor are still within the scope of protection of the present application.

Claims

1. A method for dense image description generation, characterized in that, The method comprises the following steps: acquiring an image to be processed; extracting a region of interest in the image to be processed, extracting a proposal feature of each region of interest and an adjacent feature of each region of interest adjacent to the region of interest, determining a semantic context relationship of each region of interest, determining a spatial context relationship of each region of interest according to a center position of each region of interest and a center position of each region of interest adjacent to the region of interest, and combining the semantic context relationship and the spatial context relationship to obtain a visual embedding; extracting a visual object in the image to be processed, retrieving a target triple related to the visual object from a knowledge graph, and encoding the target triple into a knowledge embedding; after the visual embedding and the knowledge embedding are mapped to the same dimension, the visual embedding and the knowledge embedding are mutually subjected to feature embedding, and a corresponding semantic description sentence is obtained through decoding after the feature embedding; wherein the obtained enhanced visual embedding and the enhanced knowledge embedding are restored to the original dimension and are spliced with original embeddings through residual connection; the enhanced visual embedding after the residual connection is projected into a language feature space through an alignment network, and then the enhanced visual embedding after the residual connection and the enhanced knowledge embedding after the residual connection are input into a text decoder to obtain a corresponding semantic description sentence.

2. The dense image description generation method of claim 1, wherein, The process of determining the semantic context relationship of each region of interest includes: splicing the adjacent features of each adjacent region of interest and the proposed features of the region of interest, determining the semantic context relationship between the adjacent features of each adjacent region of interest and the proposed features of the region of interest through a weight matrix representing a rectangular frame of the i-th adjacent region of interest and a rectangular frame of the region of interest The semantic context relationship between the adjacent features of each adjacent region of interest and the proposed features of the region of interest is determined through a weight matrix All elements in the learned weight matrix are constrained in the range of (-1, 1).

3. The dense image description generation method of claim 1, wherein, The process of determining the spatial context relationship of each region of interest includes: determining the center position of each adjacent region of interest and the center position of the region of interest performing splicing through a weight matrix representing a rectangular frame of the i-th adjacent region of interest and the rectangular frame of the region of interest between the spatial context relationship, and using an activation function all elements in the learned weight matrix are constrained in the range of [0, +∞).

4. The dense image description generation method of claim 1, wherein, The process of extracting the knowledge embedding comprises: ； wherein, is a target triple; is a non-parametric scoring function for computing similarity between an input text representation and a triple representation by a dot product; represents a process of encoding a triple using a sentence encoder represents a triple , specifically, representing a triple as a token sequence consisting of entity and relation tokens and a separator token between them, and then passing the triple tokens to a sentence encoder to obtain a triple representation; G is a set of factual triples; t is a triple defined for a particular fact. finally, the retrieved target triple is embedded and encoded into a knowledge embedding by a word segmenter.

5. The dense image description generation method of claim 1, wherein, The process of mutually subjecting the visual embedding and the knowledge embedding to feature embedding comprises: when the feature in the knowledge embedding is transmitted to the visual embedding, the knowledge embedding is taken as a key K and a value V, and the visual embedding is taken as a query Q; when the feature in the visual embedding is transmitted to the knowledge embedding, the knowledge embedding is taken as a query Q, and the visual embedding is taken as a key K and a value V.

6. A dense image description generation system, characterized by, The method comprises the following steps: an acquisition module configured to acquire an image to be processed; a visual embedding extraction module configured to extract a region of interest in the image to be processed, extract a proposal feature of each region of interest and an adjacent feature of each region of interest adjacent to the region of interest, determine a semantic context relationship of each region of interest, determine a spatial context relationship of each region of interest according to a center position of each region of interest and a center position of each region of interest adjacent to the region of interest, and combine the semantic context relationship and the spatial context relationship to obtain a visual embedding; a knowledge embedding extraction module configured to extract a visual object in the image to be processed, retrieve a target triple related to the visual object from a knowledge graph, and encode the target triple into a knowledge embedding; an alignment and decoding module configured to, after the visual embedding and the knowledge embedding are mapped to the same dimension, mutually subject the visual embedding and the knowledge embedding to feature embedding, and obtain a corresponding semantic description sentence through decoding after the feature embedding; wherein the obtained enhanced visual embedding and the enhanced knowledge embedding are restored to the original dimension and are spliced with original embeddings through residual connection; the enhanced visual embedding after the residual connection is projected into a language feature space through an alignment network, and then the enhanced visual embedding after the residual connection and the enhanced knowledge embedding after the residual connection are input into a text decoder to obtain a corresponding semantic description sentence.

7. An electronic device, comprising: A computer program product comprising a memory and a processor, and computer instructions stored on the memory and running on the processor, which, when run by the processor, complete the method of any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, A computer program product for storing computer instructions, which, when executed by a processor, complete the method of any one of claims 1-5.

9. A computer program product, characterised in that, A computer program product comprising a computer program, which, when executed by a processor, implements the method of any one of claims 1-5.

Images