Dense image description generation method, system, device, medium and program product

Abstract

The invention discloses a dense image description generation method, system and device, a medium and a program product, and relates to the technical field of dense description, and the method comprises the steps: extracting regions of interest in a to-be-processed image, extracting a proposal feature of each region of interest and an adjacent feature of each region of interest adjacent to the proposal feature, thereby determining a semantic context relationship, and generating a dense image description; determining a spatial context relationship according to the central position of each region of interest and the central position of each adjacent region of interest, and combining the semantic context relationship with the spatial context relationship to obtain visual embedding; a visual object in the to-be-processed image is extracted, a target triple related to the visual object is retrieved from the knowledge graph, and the target triple is coded into knowledge embedding; after the visual embedding and the knowledge embedding are mapped to the same dimension, the visual embedding and the knowledge embedding are mutually subjected to feature embedding, and semantic description sentences are obtained through decoding, so that the dense description generation performance is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of dense description technology, and in particular to a dense image description generation method, system, device, medium and program product. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Dense description technology covers object detection, semantic content parsing, and natural language description. With the release of large-scale description datasets and the advancement of deep learning technology, the performance of dense description has been significantly improved, and it has been widely used, including human-computer interaction, image-text matching, visual question answering, etc. Dense description goes a step further than image description and provides a richer and more detailed description of the image. In the prior art, a fully convolutional localization network (FCLN) is proposed, which includes a detector to identify all regions of interest (RoI) and generate their text descriptions.

[0004] In existing research, it can be roughly divided into two stages according to whether context information is included. The initial architecture uses Faster R-CNN to detect RoI and uses Long Short-Term Memory (LSTM) to generate text description; however, these frameworks focus mainly on RoI and ignore potentially valuable context information. Subsequent research solves this problem by integrating RoI features with global image features before text description, but the global context is too coarse. There are also methods that study fine-grained context information, such as proposing a non-local similarity graph to promote feature interaction between the target RoI and its neighboring RoIs, and revealing the close relationship between RoI and the detected object based on data statistical methods, thereby proposing an architecture that considers context information, but it still fails to effectively utilize the potential advantages of context clues. Language clues from surrounding RoIs are always valuable. When generating a description for a specific RoI, the description obtained by focusing only on the RoI without combining context information to observe the surrounding environment is incomplete and not rich.

[0005] Secondly, the Transformer-based Dense Descriptor (TDC) proposed by the Transformer architecture gives different importance to each detected RoI through the Region-Object Correlation Score Unit (ROCSU). However, the existing context modeling mechanism heavily relies on the quality of object proposals. Due to image blur or the presence of small objects, low-quality object proposals are inevitably generated, which contain less semantic information.

[0006] In addition, existing dense description methods inadvertently involve unbalanced learning during the learning process, mainly due to the uneven distribution of training data. The trained model tends to generate descriptions containing words that frequently appear in the training samples, resulting in limited vocabulary diversity and insufficient description of object attributes. To address this problem, one method proposes a dynamic word frequency histogram resampling strategy to rebalance words of different frequencies during training. However, this method is often insufficient in understanding the intrinsic reasons for certain visual attributes, such as the common color of vegetables, which is usually green. This concept of "reason" goes beyond visual features and covers higher-level common sense reasoning, which existing methods cannot accurately explain. Summary of the invention

[0007] In order to solve the above problems, the present invention proposes a dense image description generation method, system, device, medium and program product, which generate dense image description based on external knowledge graph and visual context perception, ensure the consistency of visual local description and context, and capture the attributes of visual objects and their topological relationship with other objects, effectively improving the performance of dense description generation.

[0008] In order to achieve the above object, the present invention adopts the following technical solution:

[0009] In a first aspect, the present invention provides a method for generating dense image descriptions, comprising:

[0010] Get the image to be processed;

[0011] Extract the regions of interest in the image to be processed, extract the proposed features and the adjacent features of each region of interest adjacent to it for each region of interest, so as to determine the semantic context relationship of each region of interest, determine the spatial context relationship of each region of interest according to the center position of each region of interest and the center position of each region of interest adjacent to it, and combine the semantic context relationship and the spatial context relationship to obtain visual embedding;

[0012] Extract visual objects in the image to be processed, retrieve target triples related to the visual objects from the knowledge graph, and encode the target triples into knowledge embeddings;

[0013] After mapping visual embedding and knowledge embedding to the same dimension, visual embedding and knowledge embedding perform feature embedding on each other, and after feature embedding, the corresponding semantic description sentence is obtained by decoding.

[0014] As an optional implementation, the process of determining the semantic context relationship of each region of interest includes: concatenating the adjacent features of each adjacent region of interest and the proposed features of the region of interest, and calculating the semantic context relationship of each region of interest by a weight matrix W. p Rectangular box o representing the i-th adjacent region of interesti and the rectangular box o of the region of interest p The semantic contextual relationship between them is calculated, and the activation function Tanh is used to convert the learned weight matrix W p All elements in are constrained to be in the range (-1,1).

[0015] As an optional implementation, the process of determining the spatial context relationship of each region of interest includes: i and the center position c of the region of interest p For splicing, through the weight matrix W q Rectangular box o representing the i-th adjacent region of interest i and the rectangular box o of the region of interest p The spatial contextual relationship between them is constructed, and the activation function ReLU is used to transform the learned weight matrix W q All elements in are constrained to be in the range [0,+∞).

[0016] As an optional implementation, the process of extracting knowledge embedding includes:

[0017]

[0018] Among them, t + is the target triplet; f is a non-parametric scoring function used to calculate the input text representation E through the dot product θ (x) and triples represent E θ (t) Similarity between; E θ (t) represents the use of sentence encoder E θ The process of representing a triple t is as follows: the triple is represented as a sequence of tokens, which consists of entity and relation tokens and the separation tokens between them, and then the triple tokens are passed to the sentence encoder to obtain the triple representation; G is a set of fact triplets; t is a triple defined by a specific fact;

[0019] Finally, the retrieved target triples are embedded and encoded into knowledge embeddings through the word segmenter.

[0020] As an optional implementation, the process of mutual feature embedding of visual embedding and knowledge embedding includes:

[0021] When passing features from knowledge embedding to visual embedding, the knowledge embedding is used as the key K and value V, and the visual embedding is used as the query Q; when passing features from visual embedding to knowledge embedding, the knowledge embedding is used as the query Q, and the visual embedding is used as the key K and value V.

[0022] As an optional implementation method, after feature embedding, the obtained enhanced visual embedding and enhanced knowledge embedding are restored to the original dimensions and spliced ​​with the original embedding through residual connection; the enhanced visual embedding after residual connection is projected into the language feature space through the alignment network, and then input into the text decoder with the enhanced knowledge embedding after residual connection to obtain the corresponding semantic description sentence.

[0023] In a second aspect, the present invention provides a dense image description generation system, comprising:

[0024] An acquisition module, configured to acquire an image to be processed;

[0025] A visual embedding extraction module is configured to extract regions of interest in the image to be processed, extract a proposed feature and adjacent features of each region of interest adjacent to the region of interest for each region of interest, thereby determining a semantic context relationship of each region of interest, determining a spatial context relationship of each region of interest according to a center position of each region of interest and a center position of each region of interest adjacent to the region of interest, and combining the semantic context relationship with the spatial context relationship to obtain a visual embedding;

[0026] a knowledge embedding extraction module configured to extract visual objects in the image to be processed, retrieve target triples related to the visual objects from the knowledge graph, and thereby encode the target triples into knowledge embeddings;

[0027] The alignment decoding module is configured to map the visual embedding and the knowledge embedding to the same dimension, and then perform feature embedding between the visual embedding and the knowledge embedding. After the feature embedding, the corresponding semantic description sentence is obtained through decoding.

[0028] In a third aspect, the present invention provides an electronic device comprising a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the method described in the first aspect is performed.

[0029] In a fourth aspect, the present invention provides a computer-readable storage medium for storing computer instructions, wherein when the computer instructions are executed by a processor, the method described in the first aspect is performed.

[0030] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method described in the first aspect.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] The present invention proposes a dense image description generation method based on external knowledge graph and visual context perception, which ensures the consistency of visual local description and context, and captures the properties of visual objects and their topological relationships with other objects, effectively improving the performance of dense description generation.

[0033] This paper proposes a novel common sense knowledge graph CSKG and a direct knowledge retrieval framework. This method provides prior guidance for image-to-text conversion, and only uses the representation similarity between visual objects and triples to solve the problem of answer retrieval at the semantic level; through a multimodal knowledge graph, visual objects are connected to named entities and the relationship between entities is captured by using the acquired external knowledge.

[0034] The present invention designs a context extractor CE to promote the sharing and message passing of spatial and semantic context information, and the relevant visual object features are passed to the frozen visual Transformer (ViT) to obtain visual embedding.

[0035] The present invention introduces a dual-stream interaction module DSIM to enhance the interaction between visual embedding and textual knowledge embedding, thereby improving the accuracy of region-level description.

[0036] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0038] Figure 1 A flow chart of a dense image description generation method provided in Example 1 of the present invention;

[0039] Figure 2 A schematic diagram of a visual embedding extraction module provided in Example 1 of the present invention;

[0040] Figure 3 A schematic diagram of a knowledge embedding and extraction module provided in Example 1 of the present invention;

[0041] Figure 4 A schematic diagram of an alignment decoding module provided in Embodiment 1 of the present invention;

[0042] Figure 5 This is a schematic diagram of the experimental results provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0043] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0044] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0045] It should be noted that the terms used herein are only for describing specific embodiments, and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that the terms "include" and "comprise" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0046] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0047] Terminology explanation:

[0048] Image description generation: Image description generation is a computer vision and natural language processing technology that aims to describe the image as a whole in natural language. This technology generally uses convolutional neural networks (CNNs) in deep learning to extract image features and recursive neural networks (RNNs) or Transformer models to generate descriptions.

[0049] Dense image description generation: Dense image description generation is an extension of image description generation that aims to generate detailed descriptions for multiple regions in an image rather than just a holistic description of the entire image. This technique can capture details and multiple elements in an image more finely, and is usually achieved by combining object detection and description generation techniques.

[0050] Visual context awareness: Visual context awareness refers to the computer vision system understanding and utilizing contextual information in an image or video for more accurate recognition and analysis. For example, when recognizing an object in a scene, not only the individual features of the object are considered, but also information about its surroundings and other related objects. This context awareness can improve the accuracy and robustness of the model, making it more effective in real-world applications.

[0051] Knowledge retrieval: Knowledge retrieval is an information retrieval technique that aims to extract relevant information or answers from a large knowledge base or database. It usually involves natural language processing techniques to understand the user's query and match and retrieve the most relevant knowledge fragments.

[0052] Multimodal learning: Multimodal learning is a machine learning method that combines and utilizes data from multiple modalities (such as text, images, audio, video, etc.) for training and reasoning. The core of this method is to improve the model's understanding and prediction capabilities by fusing different types of data sources.

[0053] Example 1

[0054] This embodiment provides a dense image description generation method, including:

[0055] Get the image to be processed;

[0056] Extract the regions of interest in the image to be processed, extract the proposed features and the adjacent features of each region of interest adjacent to it for each region of interest, so as to determine the semantic context relationship of each region of interest, determine the spatial context relationship of each region of interest according to the center position of each region of interest and the center position of each region of interest adjacent to it, and combine the semantic context relationship and the spatial context relationship to obtain visual embedding;

[0057] Extract visual objects in the image to be processed, retrieve target triples related to the visual objects from the knowledge graph, and encode the target triples into knowledge embeddings;

[0058] After mapping visual embedding and knowledge embedding to the same dimension, visual embedding and knowledge embedding perform feature embedding on each other, and after feature embedding, the corresponding semantic description sentence is obtained by decoding.

[0059] like Figure 1 As shown, this embodiment proposes a dense image description network (EKCA-Cap) based on external knowledge graph and visual context perception. The overall framework of the network includes a visual embedding extraction module, a knowledge embedding extraction module and an alignment decoding module. In addition, multiple pre-trained frozen models are used to extract features. The multiple pre-trained frozen models include a frozen Faster R-CNN model, a frozen ViT model, a frozen DE (Detection Everything) model, a frozen word segmenter embedding, an alignment network and a text decoder.

[0060] Specifically:

[0061] In the visual embedding extraction module, first, a CNN model is used to learn low-level semantic features from the input image to be processed, using the Faster R-CNN model; then, the semantic features are passed through the Region Proposal Network (RPN) to generate a set of regions of interest; then, a contextual feature extractor is introduced to generate visual features related to each region of interest RoI, and the visual features are input into the ViT model to extract rich and fine-grained visual embeddings.

[0062] In the knowledge embedding extraction module, first, the DE model is used to identify all visual objects in the input image to be processed; then, the knowledge related to the visual objects is retrieved from the common sense knowledge graph and these knowledge words are encoded into knowledge embeddings.

[0063] In the alignment decoding module, the generated visual embedding and knowledge embedding exchange information through a two-stream interaction module to solve the misalignment problem generated by each module; then the visual embedding is projected into the language feature space through the alignment network, and then input into the text decoder together with the knowledge embedding to generate a dense description.

[0064] The overall framework of the EKCA-Cap network is described in detail below.

[0065] 1. Visual embedding extraction module.

[0066] Define the region of interest in the image to be processed as M = {M i |i=1,2,...,N}, the global area (i.e. the entire image) is M * , the adjacent region of interest is M in =M / M i , where M in is the nth adjacent region of interest to the ith region of interest. i , its proposed features, adjacent features and global features are represented as V ip 、V in and V ig .

[0067] Among them, the adjacent feature V in V in =f(M i ,M in ), where f(·) represents the visual context feature extraction module, which extracts features through spatial context and visual context respectively.

[0068] Proposed Features V p It is initially obtained through a single-layer convolutional network. Specifically:

[0069] The global region is input into CenterNet to extract key points. CenterNet is an anchor-free detector. In order to improve the recall rate, this step is slightly modified. Since the context information is merged in the low-resolution feature map, it is easier to roughly locate the object. Following the process of CenterNet, key point features c with different confidence scores are obtained. i and visual features c p Since the confidence proposals are related to neighboring objects that contain rich contextual information, these neighboring objects are used to assist the proposed features v p Description generated.

[0070] In this embodiment, the visual context is modeled by constructing a relationship graph G = (V, E, g); wherein v i ∈V is the neighboring features of the adjacent region of interest, g is the global region M * The eigenvector of ip ∈E represents v i and v p The relationship between.

[0071] In this embodiment, e ip Indicates the relationship between scene and object, and its constant value is set to 1; specifically:

[0072] e ip =ReLU(W q [c i ,c p ])*Tanh(W p [v i ,v p ]);

[0073] Among them, c i and c p Rectangular box o representing the i-th adjacent region of interest i and the rectangular box o of the region of interest p The central location of W q and W p is a learnable weight matrix.

[0074] like Figure 2 As shown, the final context relationship is obtained based on the merging of the semantic context relationship and the spatial context relationship; specifically:

[0075] (1) Learning semantic contextual relationships: First, the adjacent features v of each adjacent region of interest are i and the proposed features v of the region of interest p Then, through the weight matrix W p , to indicate o i and pThe semantic contextual relationship between them; In addition, the activation function Tanh is used to transform the learned weight matrix W p All elements in are constrained to be in the range (-1,1).

[0076] (2) Learning of spatial contextual relationships: First, the center position c of each adjacent region of interest is i and the center position c of the region of interest p Then, through the weight matrix W q , to indicate o i and p The spatial context relationship between them; since the value of the spatial context relationship is non-negative, the activation function ReLU is used to convert the learned weight matrix W q All elements in are constrained to be in the range [0,+∞).

[0077] (3) Multiply the semantic context relationship and the spatial context relationship to obtain the scene object feature e ip , the scene object feature e ip Input into the ViT model to extract the final visual embedding F v .

[0078] 2. Knowledge embedding extraction module.

[0079] Let E be the set of entities and R be the set of relations. A particular fact is defined as a triple: t = (e h ,r,e t )∈E×R×E, where e h and e t are the head entity and the tail entity respectively, and r is the relationship between them. In addition, the knowledge graph is defined to represent a set of fact triples:

[0080] It is worth noting that this knowledge graph is a valuable source of knowledge for many natural language applications. However, the input of most natural language tasks is usually represented as a sequence of tokens: x = [w1, w2, ..., w |x| ].

[0081] Assume that given input x, t + is a target triplet to be retrieved. The goal of the traditional fact retrieval process is usually formalized as the following three consecutive tasks:

[0082]

[0083] in, is the mention detection model, m is the entity mention detected in the input x, and p φ (e|m,x) is the entity disambiguation model, p θ(t|e,x,G) is a relational classification model.

[0084] However, this three-step approach has some limitations. First, they are prone to accumulating errors. Second, due to its decomposed structure, it is difficult to train these three sub-modules in an end-to-end manner while providing labels for the training of each sub-module.

[0085] In order to solve the above challenges of existing knowledge graph fact retrieval methods, this embodiment proposes a direct knowledge retrieval framework, such as Figure 3 As shown. The goal of this embodiment is to use a single sentence encoder model E θ Simplifying the process without introducing extra variables (such as m and e), it looks like this:

[0086]

[0087] Where f is a non-parametric scoring function used to calculate the input text representation E through the dot product θ (x) and triples represent E θ (t) Similarity between; E θ (t) represents the use of sentence encoder E θ The process of representing a triple t is as follows: First, the triple t is represented as a tag sequence: t = [w1, w2, ..., w |t| ], which consists of entity and relation tags and the separator between them (i.e., special tag [SEP]); then, the triple tag is passed to E θ , to obtain the triple representation; finally, the target triple representation is encoded into knowledge embedding through the word segmenter embedding.

[0088] 3. Alignment decoding module.

[0089] Since the various common sense elements contained in the knowledge graph derived from the common sense knowledge graph may not be directly related to the current visual context, this brings up the challenge of aligning multiple knowledge embeddings with a single visual embedding, a phenomenon known as the "variation problem" of common sense knowledge. To address this challenge, this embodiment introduces a two-stream interaction module to enhance the information exchange between visual embeddings and knowledge embeddings, thereby improving the alignment effect.

[0090] like Figure 4 As shown, the dual-stream interaction module includes: an adaptation layer and a cross-modal interaction module.

[0091] Specifically:

[0092] The adaptation layer consists of a single linear layer to embed the visual v Or knowledge embedding F kto a low-dimensional latent space, or map them back to the original feature space. In the two-stream interaction module, first, the visual embedding and the knowledge embedding are mapped to the same latent space through an adaptation layer.

[0093] Secondly, the features in the knowledge embedding are passed to the visual embedding through the cross-modal interaction layer and the feed-forward layer, where the knowledge embedding is used as the key K and value V, and the visual embedding is used as the query Q; at the same time, the features in the visual embedding are passed to the knowledge embedding through the cross-modal interaction layer and the feed-forward layer, where the knowledge embedding is used as the query Q, and the visual embedding is used as the key K and value V.

[0094] Then, after feature embedding, the obtained enhanced visual embedding and enhanced knowledge embedding are restored to their original feature space and concatenated with the original embedding through residual connection;

[0095] Finally, the enhanced visual embedding after residual connection is projected into the language feature space through the alignment network, and then input into the text decoder together with the enhanced knowledge embedding after residual connection to obtain the corresponding semantic description sentence. Among them, the alignment network adopts this structure to achieve more effective embedding alignment.

[0096] 4. Experiments and results.

[0097] The Visual Genome (VG), VG-COCO and RefCOCOg datasets are used as evaluation benchmarks, and METEOR and CIDEr are used as evaluation metrics. These experiments are conducted using four NVIDIA3090 graphics processors running Ubuntu 18.04.

[0098] (1) mAP comparison: The method EKCA-Cap of this embodiment is compared with other baseline methods, as shown in Table 1. The results show that EKCA-Cap achieves 15.82% mAP on the VG V1.0 dataset, 15.65% on the VG V1.2 dataset, and 18.20% on the VG-COCO dataset, which is significantly better than the existing state-of-the-art methods (SOTA).

[0099] Specifically, compared with the SOTA method DCMSTRD, the method of this embodiment achieves a mAP gain of 2.21 on the VG V1.2 dataset. These performance improvements are mainly attributed to the advantages of the contextual feature extractor and the integration of textual knowledge in the common sense knowledge graph associating visual objects, which enhances the refinement of subtitles, including object attributes and topological relationship subtitle generation.

[0100] To further illustrate the capability of the model EKCA-Cap in generating dense captions, the influence of visual localization is ignored. When the real bounding box is provided, the mAP of EKCA-Cap on VG V1.0, VG V1.2 and VG-COCO reaches 46.65%, 44.32% and 49.35% respectively, which is 12.45% higher than BLIP2 on VG-COCO.

[0101] Table 1 mAP indicators of the model on VG and VG-COCO datasets

[0102]

[0103]

[0104] (2) Comparison of Meteor and Cider metrics: The comparison experiment used the same settings as the previous study, as shown in Table 2. On the RefCOCOg dataset, EKCA-Cap significantly outperforms these methods, with the METEOR score increasing from 16.6 to 17.2 and the CIDEr score increasing from 108.3 to 113.4, which is a significant improvement over the previous SOTA method Osprey. This improvement is mainly attributed to the CE module in EKCA-Cap, which provides a wider range of context through the text context of other neighboring RoIs during caption generation. In addition, CSKG is able to balance the frequency of word usage, resulting in the above excellent results.

[0105] This embodiment further evaluates the EKCA-Cap method on the VG V1.2 dataset. The METEOR score results are shown in the fourth column of Table 2, and the CIDEr score results are shown in the fifth column. Obviously, the method EKCA-Cap proposed in this embodiment has a relative improvement of 2.3 in METEOR score and 23.1 in CIDEr score compared to the Alpha-CLIP+LLaVA method. It is worth noting that on the VG V1.2 dataset, the performance gap between the method proposed in this embodiment and the SOTA method is smaller than that on the RefCOCOg dataset. This difference may be due to the larger scale of the VG V1.2 dataset, which contains more images with complex scenes and subtitles. Even under the guidance of CSKG and CE, generating subtitles is more challenging.

[0106] Table 2 METEOR and CIDEr indicators of the model on VG and RefCOCOg datasets

[0107]

[0108]

[0109] (3) The effectiveness of the components CSKG, CE and DSIM proposed in this embodiment is demonstrated through ablation experiments. DSIM aims to maximize the interaction between visual and textual knowledge modalities. Therefore, when CSKG is removed, DSIM should also be omitted. By introducing CSKG and DSIM, the mAP on the VG V1.2 dataset is improved by 2.06% (from 9.52% to 11.58%) (Table 3, rows 1-2). Similarly, the introduction of the context extractor (CE) improves the mAP on the VG V1.2 dataset by 4.8% (from 9.52% to 14.32%) (Table 3, rows 1-3). On the VG-COCO dataset, the introduction of CSKG and DSIM brings a 3.05% improvement, while the introduction of CE brings a 4.94% improvement. These significant improvements highlight the importance of CSKG, CE and DSIM. In particular, the CE module integrates multi-scale context and contextual cues between individual regions in region generation. Contextual clues, including global and neighborhood information, contribute significantly to improving the final performance.

[0110] Table 3 Ablation experiment

[0111]

[0112] The qualitative results and analysis of the method of this embodiment are presented below to provide a subjective evaluation of the method of this embodiment. Figure 5 Four examples of dense captioning results are shown in Figure 1. Visualize all RoIs and their descriptions. Specifically, Figure 5 (a) and (c) in Figure 2 show examples from the VG-COCO dataset. Figure 5 (b) and (d) in Figure 2 show examples from the VG V1.2 dataset. Figure 5 As shown in (a) and (b) in Figure 1, the EKCA-Cap proposed in this embodiment shows high quality in RoI localization and caption generation. It is worth noting that the model of this embodiment demonstrates proficient grammar use, and most of the generated sentences are readable and meet correct English grammar and human understanding standards. In addition, the regional captions include attribute information of visual targets, such as "gree grass", "white shirt" and "blue jeans" in (a), and "brown wooden floor" in (b), which is mainly due to CSKG, which provides information about common object attributes and their relationships.

[0113] The EKCA-Cap method proposed in this embodiment can effectively detect RoIs of different scales. Figure 5In (a), it accurately identifies and describes the RoI containing two people standing on the grass, each with a different scale. In (b), the method of this embodiment recognizes many kitchen-related objects, such as dishwashers and microwaves, and the generated regional captions are mostly kitchen-related. This ability is attributed to the guidance of CE, which provides language context clues from the surrounding RoIs during the caption generation process.

[0114] This example also shows the dense captioning results of the baseline model COCG, as shown in (c) and (d). The method proposed in this example is able to recognize small objects, such as the red Frisbee clearly identified in the first example. This ability stems from the guidance of CSKG, which provides common sense understanding. It is clear from the context that a man is bending over and handing something to another person on the grass. Figure 5 In the second example shown in (b) of Figure 1, the CSKG and CE proposed in this embodiment are able to extract rich information from the image. Specifically, based on the context, it can be judged that the environment is a kitchen. Therefore, captions such as "a black dishwasher", "a black microwave", and "a clear plastic bottle" are generated, covering objects and their attributes related to the kitchen. For the baseline model COCG, the results lack relevant descriptions of the kitchen, such as Figure 5 As shown in (d) in .

[0115] Example 2

[0116] This embodiment provides a dense image description generation system, including:

[0117] An acquisition module, configured to acquire an image to be processed;

[0118] A visual embedding extraction module is configured to extract regions of interest in the image to be processed, extract a proposed feature and adjacent features of each region of interest adjacent to the region of interest for each region of interest, thereby determining a semantic context relationship of each region of interest, determining a spatial context relationship of each region of interest according to a center position of each region of interest and a center position of each region of interest adjacent to the region of interest, and combining the semantic context relationship with the spatial context relationship to obtain a visual embedding;

[0119] a knowledge embedding extraction module configured to extract visual objects in the image to be processed, retrieve target triples related to the visual objects from the knowledge graph, and thereby encode the target triples into knowledge embeddings;

[0120] The alignment decoding module is configured to map the visual embedding and the knowledge embedding to the same dimension, and then perform feature embedding between the visual embedding and the knowledge embedding. After the feature embedding, the corresponding semantic description sentence is obtained through decoding.

[0121] In this embodiment, the process of determining the semantic context relationship of each region of interest includes: concatenating the adjacent features of each adjacent region of interest and the proposed features of the region of interest, and using the weight matrix W p Rectangular box o representing the i-th adjacent region of interest i and the rectangular box o of the region of interest p The semantic contextual relationship between them is calculated, and the activation function Tanh is used to convert the learned weight matrix W p All elements in are constrained to be in the range (-1,1).

[0122] In this embodiment, the process of determining the spatial context relationship of each region of interest includes: i and the center position c of the region of interest p For splicing, through the weight matrix W q Rectangular box o representing the i-th adjacent region of interest i and the rectangular box o of the region of interest p The spatial contextual relationship between them is constructed, and the activation function ReLU is used to transform the learned weight matrix W q All elements in are constrained to be in the range [0,+∞).

[0123] In this embodiment, the process of extracting knowledge embedding includes:

[0124]

[0125] Among them, t + is the target triplet; f is a non-parametric scoring function used to calculate the input text representation E through the dot product θ (x) and triples represent E θ (t) Similarity between; E θ (t) represents the use of sentence encoder E θ The process of representing a triple t is as follows: the triple is represented as a sequence of tokens, which consists of entity and relation tokens and the separation tokens between them, and then the triple tokens are passed to the sentence encoder to obtain the triple representation; G is a set of fact triplets; t is a triple defined by a specific fact;

[0126] Finally, the retrieved target triples are embedded and encoded into knowledge embeddings through the word segmenter.

[0127] In this embodiment, the process of mutual feature embedding of visual embedding and knowledge embedding includes:

[0128] When passing features from knowledge embedding to visual embedding, the knowledge embedding is used as the key K and value V, and the visual embedding is used as the query Q; when passing features from visual embedding to knowledge embedding, the knowledge embedding is used as the query Q, and the visual embedding is used as the key K and value V.

[0129] In this embodiment, after feature embedding, the obtained enhanced visual embedding and enhanced knowledge embedding are restored to the original dimensions and spliced ​​with the original embedding through residual connection; the enhanced visual embedding after residual connection is projected into the language feature space through the alignment network, and then input into the text decoder with the enhanced knowledge embedding after residual connection to obtain the corresponding semantic description sentence.

[0130] It should be noted that the above modules correspond to the steps described in Example 1, and the examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above Example 1. It should be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer executable instructions.

[0131] In further embodiments, there is also provided:

[0132] An electronic device includes a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the method described in Embodiment 1 is performed. For the sake of brevity, it will not be described in detail here.

[0133] It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0134] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0135] A computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by a processor, the method described in Example 1 is completed.

[0136] The method in Example 1 can be directly embodied as a hardware processor, or a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it is not described in detail here.

[0137] A computer program product includes a computer program, and when the computer program is executed by a processor, the method described in embodiment 1 is implemented.

[0138] The present invention also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer executable instructions, such as instructions included in a program module, which are executed in a device on a real or virtual processor of the target to perform the process / method as described above. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform specific tasks or implement specific abstract data types. In various embodiments, the functions of program modules can be combined or divided between program modules as needed. Machine executable instructions for program modules can be executed in local or distributed devices. In distributed devices, program modules can be located in local and remote storage media.

[0139] The computer program code for implementing the method of the present invention can be written in one or more programming languages. These computer program codes can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the computer or other programmable data processing device, causes the function / operation specified in the flow chart and / or block diagram to be implemented. The program code can be executed completely on a computer, partially on a computer, as an independent software package, partially on a computer and partially on a remote computer or completely on a remote computer or server.

[0140] In the context of the present invention, computer program codes or related data may be carried by any appropriate carrier to enable a device, apparatus or processor to perform the various processes and operations described above. Examples of carriers include signals, computer readable media, etc. Examples of signals may include electrical, optical, radio, acoustic or other forms of propagation signals, such as carrier waves, infrared signals, etc.

[0141] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0142] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A dense image description generation method, characterized in that: include: Get the image to be processed; Extract the regions of interest in the image to be processed, extract the proposed features and the adjacent features of each region of interest adjacent to it for each region of interest, so as to determine the semantic context relationship of each region of interest, determine the spatial context relationship of each region of interest according to the center position of each region of interest and the center position of each region of interest adjacent to it, and combine the semantic context relationship and the spatial context relationship to obtain visual embedding; Extract visual objects in the image to be processed, retrieve target triples related to the visual objects from the knowledge graph, and encode the target triples into knowledge embeddings; After mapping visual embedding and knowledge embedding to the same dimension, visual embedding and knowledge embedding perform feature embedding on each other, and after feature embedding, the corresponding semantic description sentence is obtained by decoding.

2. A dense image description generation method as claimed in claim 1, characterized in that: The process of determining the semantic context of each region of interest includes: concatenating the adjacent features of each adjacent region of interest and the proposed features of the region of interest, and using the weight matrix W p Rectangular box o representing the i-th adjacent region of interest i and the rectangular box o of the region of interest p The semantic contextual relationship between them is calculated, and the activation function Tanh is used to convert the learned weight matrix W p All elements in are constrained to be in the range (-1,1).

3. A dense image description generation method as claimed in claim 1, characterized in that: The process of determining the spatial context of each region of interest includes: taking the center position c of each adjacent region of interest as i and the center position c of the region of interest p For splicing, through the weight matrix W q Rectangular box o representing the i-th adjacent region of interest i and the rectangular box o of the region of interest p The spatial contextual relationship between them is constructed, and the activation function ReLU is used to transform the learned weight matrix W q All elements in are constrained to be in the range [0,+∞).

4. A dense image description generation method as claimed in claim 1, characterized in that: The process of extracting knowledge embedding includes: Among them, t + is the target triplet; f is a non-parametric scoring function used to calculate the input text representation E through the dot product θ (x) and triples represent E θ (t) Similarity between; E θ (t) represents the use of sentence encoder E θ The process of representing a triple t is as follows: the triple is represented as a sequence of tokens, which consists of entity and relation tokens and the separation tokens between them, and then the triple tokens are passed to the sentence encoder to obtain the triple representation; G is a set of fact triplets; t is a triple defined by a specific fact; Finally, the retrieved target triples are embedded and encoded into knowledge embeddings through the word segmenter.

5. A dense image description generation method as claimed in claim 1, characterized in that: The process of mutual feature embedding between visual embedding and knowledge embedding includes: When passing features from knowledge embedding to visual embedding, the knowledge embedding is used as the key K and value V, and the visual embedding is used as the query Q; when passing features from visual embedding to knowledge embedding, the knowledge embedding is used as the query Q, and the visual embedding is used as the key K and value V.

6. A dense image description generation method as claimed in claim 1, characterized in that: After feature embedding, the enhanced visual embedding and enhanced knowledge embedding are restored to the original dimensions and concatenated with the original embedding through residual connection. The enhanced visual embedding after residual connection is projected into the language feature space through the alignment network, and then input into the text decoder with the enhanced knowledge embedding after residual connection to obtain the corresponding semantic description sentence.

7. A dense image description generation system, characterized in that: include: An acquisition module, configured to acquire an image to be processed; A visual embedding extraction module is configured to extract regions of interest in the image to be processed, extract a proposed feature and adjacent features of each region of interest adjacent to the region of interest for each region of interest, thereby determining a semantic context relationship of each region of interest, determining a spatial context relationship of each region of interest according to a center position of each region of interest and a center position of each region of interest adjacent to the region of interest, and combining the semantic context relationship with the spatial context relationship to obtain a visual embedding; a knowledge embedding extraction module configured to extract visual objects in the image to be processed, retrieve target triples related to the visual objects from the knowledge graph, and thereby encode the target triples into knowledge embeddings; The alignment decoding module is configured to map the visual embedding and the knowledge embedding to the same dimension, and then perform feature embedding between the visual embedding and the knowledge embedding. After the feature embedding, the corresponding semantic description sentence is obtained through decoding.

8. An electronic device, characterized in that: The method comprises a memory and a processor and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the method according to any one of claims 1 to 6 is completed.

9. A computer-readable storage medium, characterized in that: Used to store computer instructions, which, when executed by a processor, complete the method described in any one of claims 1 to 6.

10. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images