Image subtitle algorithm based on structured semantic extraction and geometric feature fusion

Abstract

The invention provides an image subtitle algorithm based on structured semantic extraction and geometric feature fusion. The image subtitle algorithm comprises a visual semantic feature extraction module and a subtitle generation module, the visual semantic feature extraction module comprises a regional feature, a network feature, a structured semantic feature and the subtitle generation module, firstly, the most similar text sentence is retrieved from an image through a CLIP model, concept semantics and attribute features are extracted, and after Top-K text features are retrieved through cosine similarity, multi-level clustering is carried out, and finally, the subtitle generation module is used for generating subtitles; and the isolation problem of semantic information extraction is fundamentally solved. And secondly, gradually fusing the grid features, the regional features and the structured semantic features by designing a geometric perception semantic enhancement encoder, and introducing geometric coordinate information in the fusion process, thereby enhancing the expression ability of the structured semantic features. According to the process, the modeling capability of the model for the image space relationship is remarkably enhanced, and the understanding of global semantics is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and natural language processing, and particularly relates to an image captioning algorithm based on structured semantic extraction and geometric feature fusion. BACKGROUND

[0002] Image captioning generation is an important research direction in the field of computer vision and natural language processing, which aims to automatically generate natural language texts describing image content through algorithms. In current research, most methods only rely on a single visual feature (such as regional features or grid features) for caption generation, often failing to fully express the complex semantics and spatial relationships between objects in the image.

[0003] In addition, existing methods often ignore the structured semantic relationships between concepts or attributes, treating each concept or attribute as an independent element without considering their inherent associations and hierarchical structures. This processing approach results in captions that lack semantic coherence and contextual relevance, failing to comprehensively and accurately describe objects and their relationships in the image. SUMMARY

[0004] The present application aims to address the shortcomings of the prior art and provides an image captioning algorithm based on structured semantic extraction and geometric feature fusion.

[0005] To achieve the above-mentioned purposes, the present application adopts the following technical solutions: The present application provides an image captioning algorithm based on structured semantic extraction and geometric feature fusion, comprising a visual semantic feature extraction module and a caption generation module. The visual semantic feature extraction module comprises: Regional features: used to capture local details and key object information in the image; Network features: provide global contextual semantics; Structured semantic features: through the introduction of CLIP retrieval and multi-level clustering, semantic representations with hierarchical relationships are extracted; The caption generation module: based on the extracted multi-modal features, the features are decoded and integrated through a deep neural network, transforming into fluent and semantically accurate natural language descriptions.

[0006] Further, the extraction process of regional features, network features, and structured semantic features in the visual semantic feature extraction module is as follows: Grid features and network geometric features: the extraction of grid category features is achieved by dividing the image into fixed-size grids and using a neural network model to extract contextual features and coordinate information for each network from each grid; Region feature and region geometry feature: the extraction of region feature is realized through a target detector, which analyzes the image in regions, extracts the feature representation of each object, and the coordinate quadruple of the object; Structured semantic feature: using the CLIP model to perform cross-modal retrieval on the image, and obtaining the Top-K text features most relevant to the image from the text corpus; using a multi-level clustering algorithm to cluster the extracted text features, constructing a hierarchical structured semantic representation, and capturing the semantic association and hierarchical relationship between concept semantics.

[0007] Further, the subtitle generation module comprises: Geometry-aware semantic reinforced encoder: by gradually fusing grid features, region features and structured semantic features, and introducing geometric coordinate information in the fusion process, the expression ability of structured semantic features is strengthened; Adaptive weighted feature decoder: using the fused global feature representation, together with the grid features and the grid features of the fused region features, through weighted fusion, and generating a word-by-word subtitle describing the image content through autoregressive decoding, in the decoding process, dynamically balancing the contribution of visual and semantic information, to ensure that the generated subtitle is semantically coherent and accurately reflects the image content and spatial relationship.

[0008] Further, the neural network model is used as a feature extractor to obtain image multi-scale features and global context information, specifically: Given an image First, resize the image to , and divide it into single non-overlapping patches, each with a resolution of , convert each patch to a feature vector through a prior embedding, and form an initial feature sequence as: (1) ; Where, is the initial feature sequence of the image I; is the feature vector of the Nth patch; is the feature vector of the th patch; is a real number space; is the feature dimension; The feature sequence passes through the hierarchical network of the neural network model to extract multi-scale features, in each layer, the sliding window attention mechanism and the cross-window attention mechanism are applied to capture local and global relationships, update the feature representation, and the formula is: (2) ; Simultaneously, at each stage, except for the last layer, the spatial dimension of the feature map is downsampled by a factor of 2 through the Patch Merging operation, thereby generating a multi-scale feature map, represented as: (3); in, Multi-scale feature maps; Total number of floors; The height of the image; For the first The feature dimension of the layer; Geometric coordinates are calculated using the resolution of the input image and the size of the multi-scale feature map. ,in, For the first The region's geometric coordinates; use and Geometric features of image regions and regional features.

[0009] Furthermore, the decoder portion of the deformable DETR is used to extract the region features and predict the coordinates of the detection box, specifically: Multi-scale feature maps extracted by neural network modules ,in, Indicates the first The feature maps of the layers have different spatial resolutions and number of channels; Initial feature vectors for learning , where each vector It is the first 1 eigenvector It is the total number of eigenvectors; In the attention module of the decoder, the object queries each vector. and convolutional feature maps They interact with each other, specifically as follows: Convolutional feature maps Each vector serves as both a key and a value. As a query, interaction is achieved through a multi-scale variable attention mechanism. The decoder consists of six layers of multi-scale variable attention modules, and the output is... ,in, To represent the object query features updated by the attention mechanism at each layer; to represent the object query features updated by the attention mechanism at each layer; in the last layer of the decoder, the output is... As the final regional feature, it is represented as: (4); in, for Indicates the first a region feature vector, is the number of region features extracted by DETR; is the dimension of each region feature vector; obtained by linear layer processing , in the form of , as the geometric features corresponding to the region features.

[0010] Further, the structured semantic features are constructed, specifically: Given an input image , the image is divided into subgraph sets, each subgraph represents a local region of the image; For the entire image, cross-modal retrieval is performed using the CLIP model to obtain words that are most relevant to the global semantics of the image, where each word corresponds to a text feature vector; At the subgraph level, independent cross-modal retrieval is performed on each subgraph using the CLIP model to obtain words related to each subgraph, where is the number of words related to the th subgraph; and the semantic information of the whole graph is gradually compensated through this process until non-repeating words are accumulated from all subgraphs; For the obtained word set , K-Means clustering algorithm is used to divide these words into clusters to obtain the first layer of prototype sets , where each prototype represents a group of semantically similar words, ; each prototype represents a group of semantically similar words; then the first layer of prototype sets is further clustered to obtain the second layer of prototype sets , where is the number of clusters in the second layer, and each prototype represents a group of more extensive concepts in semantics, and the operation is performed by looping to cluster higher-level prototype sets layer by layer until the final hierarchical semantic representation of the image is constructed, expressed as: (5); where Cluster(X, K) represents K-Means clustering on the set X to generate K clusters, is the number of clusters of the i-th layer; the obtained prototypes of each layer represent the multi-level semantic information of the image, the high-level prototype represents the main body or category information of the image; the low-level prototype represents the specific details or local features of the image.

[0011] Further, the specific process of the geometry-aware semantic reinforced encoder is as follows: To extract the position information, the region geometry feature and the grid geometry feature are calculated first, and the absolute position feature and the relative position feature are calculated respectively. For the object region, the absolute position encoding is as follows: (6); where Norm represents normalization, and represent the top-left corner and the bottom-right corner of the bounding box respectively; is the size of the entire image; is an embedding parameter matrix; The absolute position information of the grid feature is different from the region feature. Since each image has fixed coordinate information, the trigonometric function is used to encode in a fixed way, and the one-dimensional sine and cosine embedding is used to establish the absolute position relationship of the network feature, which is represented as: (7); (8); where is the number of the image grid in the row or column direction; when is even, the sine function is used to encode the position information; when is odd, the cosine function is used to encode; is , the constant 10000 is used to regulate the change rate of the sine and cosine functions at different frequencies, and different dimensions of encoding use different frequencies; The absolute position encoding of the grid is as follows: G A B ( i , j ) = [ E i , E j ] (9); where and are the row index and the column index of the grid respectively, to obtain the absolute position feature of the region and the absolute position feature of the grid; ​​​respectively represent the absolute position encoding of the image grid in the row direction and the column direction; For any two bounding boxes and , the relative position vector is calculated using the geometric coordinate information , and the formula is: λ = [ l o g ( | c x m − c x n | w m ) , l o g ( | c y m − c y n | h m ) , l o g ( w n w m ) , l o g ( h n h m ) , I o U ( m , n ) ] (10); wherein, and are the coordinates of the center point of the bounding box; and are the height and width of the bounding box; is the intersection over union of the bounding boxes and ; The relative position encoding is embedded into the high-dimensional encoding, and the formula is: (11); (12); wherein, is the relative position feature of the region feature; is the relative position feature of the network feature.

[0012] Further, the specific process of the adaptive weighted feature decoder is: In the adaptive weighted feature decoder, the input sequence represents the language feature in the autoregressive decoding stage; The decoding process produces interaction information through a masked self-attention mechanism, which is used to model the dependency between language features, and the specific formula is: (13); wherein, is a masked multi-head self-attention; is the interaction information; The interaction information is sent to three cross-modal attention modules to interact with the network feature , the region feature , and the structured semantic feature , and extract multi-modal information; For the first cross-modal attention module, the interaction information serves as the query vector, the network feature serves as the key and value, and the attention weight between the query vector and the region feature is: (14); In the second cross-modal attention module, the interaction information As the query vector, the region feature As the key and value, the generated attention weight , the formula is: (15); In the third cross-modal attention module, the interaction information As the query vector, the structured semantic feature As the key and value, the generated attention weight is: (16); Each group of attention weights generated subsequently and the query vector are spliced into a combined vector, and normalized by a Sigmoid function to obtain a probability distribution , , The formula is: c i r = S i g m o i d ( W r [ a i r ; x i * ] + b r ) (17); c i g = S i g m o i d ( W g [ a i g ; x i * ] + b g ) (18); c i s = S i g m o i d ( W s [ a i s ; x i * ] + b s ) (19); Among them, , , , , , is a learnable parameter; the probability distribution , , is multiplied by the corresponding attention weight , , , and the final output feature is calculated by a normalization layer, the formula is: (20); Among them, is layer normalization; is element-wise multiplication; is a conversion matrix.

[0013] The beneficial effects of the present application are: firstly, the most similar text sentences are retrieved from the image by the CLIP model, the concept semantics and attribute features are extracted, and after the Top-K text features are retrieved by the cosine similarity, multi-level clustering is performed, which fundamentally solves the isolated problem of semantic information extraction. Secondly, by designing a geometric perception semantic reinforcement encoder, gradually fusing grid features, regional features and structured semantic features, and introducing geometric coordinate information in the fusion process, the expression ability of the structured semantic features is strengthened. This process significantly enhances the modeling ability of the model for image spatial relationships and improves the understanding of global semantics. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 A flowchart of an image captioning algorithm based on structured semantic extraction and geometric feature fusion; Figure 2 A structured semantic feature extraction process; Figure 3 A caption generation module structure. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical scheme and advantages of the present application clearer, further detailed description of the present application will be made below with reference to the drawings. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0016] Please refer to Figure 1 An image captioning algorithm based on structured semantic extraction and geometric feature fusion, comprising a visual semantic feature extraction module and a caption generation module; The visual semantic feature extraction module comprises: Regional features: used to capture local details and key object information in the image; Network features: provide global context semantics; Structured semantic features: by introducing CLIP retrieval and multi-level clustering, semantic representations with hierarchical relationships are extracted; The caption generation module: based on the extracted multi-modal features, the features are decoded and integrated by a deep neural network, and converted into fluent and semantically accurate natural language descriptions.

[0017] Specifically, in the caption generation module, the global context features, local features, and structured semantic relationships are integrated, and the geometric coordinate information of the visual information is introduced, so that the produced caption can not only describe the main objects and behaviors in the image, but also reflect the spatial relationship between objects and semantic coherence.

[0018] The extraction process of the regional features, network features and structured semantic features in the visual semantic feature extraction module is: Grid features and network geometry features: The extraction of grid category features is achieved by dividing the image into fixed-size grids and using a neural network model to extract context features from each grid and the coordinates of each network; Region features and region geometry features: The extraction of region features is achieved by a target detector that analyzes the image region by region, extracts the feature representation of each object, and the coordinate quadruple of the object; Structured semantic features: The CLIP model is used for cross-modal retrieval of images, and the Top-K text features most relevant to the image are obtained from the text corpus; a multi-level clustering algorithm is used to cluster the extracted text features, constructing a hierarchical semantic representation that captures the semantic associations and hierarchical relationships between concept semantics.

[0019] The subtitle generation module includes: Geometric perception semantic reinforcement encoder: by gradually fusing grid features, region features and structured semantic features, and introducing geometric coordinate information in the fusion process, the expression ability of structured semantic features is strengthened; this process significantly improves the modeling ability of image spatial relationships and the understanding of global semantics.

[0020] Adaptive weighted feature decoder: using the fused global feature representation, together with the grid features of the grid features and the fused region features, through weighted fusion, and through autoregressive decoding to generate word-by-word subtitles describing the image content. In the decoding process, the contributions of visual and semantic information are dynamically balanced to ensure that the generated subtitles are semantically coherent and accurately reflect the image content and spatial relationships.

[0021] The main advantage of the adaptive weighted feature decoder is its ability to flexibly adjust the importance of different features (such as region features, grid features and structured semantic features) in the decoding process. By introducing a dynamic weighting mechanism, the decoder can adaptively adjust the contribution of each feature according to the intermediate state of each layer and the context information in the generation process. This allows the model to more accurately combine visual information and text semantics when processing multi-modal information, thereby improving the quality and consistency of the generated text.

[0022] In the image captioning task, in addition to the features of individual blocks, the relative positional relationships between image blocks are also very important.

[0023] The neural network model is used as a feature extractor to obtain image multi-scale features and global context information, specifically: Given an image First, resize the image to size, and divide it into single non-overlapping small blocks, each with a resolution of By pre-embedding, each of the small blocks is converted into a feature vector, resulting in the following initial feature sequence: (1); in, This is the initial feature sequence of image I; The feature vector of the Nth small block; For the first Feature vectors of small blocks; for 3D real space; For feature dimensions; The feature sequence is processed through a hierarchical network of a neural network model to extract multi-scale features. In each layer, sliding window attention and cross-window attention mechanisms are applied to capture local and global relationships and update the feature representation. The formula is as follows: (2); Simultaneously, at each stage, except for the last layer, the spatial dimension of the feature map is downsampled by a factor of 2 through the Patch Merging operation, thereby generating a multi-scale feature map, represented as: (3); in, Multi-scale feature maps; Total number of floors; The height of the image; For the first The feature dimension of the layer; Geometric coordinates are calculated using the resolution of the input image and the size of the multi-scale feature map. ,in, For the first The region's geometric coordinates; use and Geometric features of image regions and regions.

[0024] Region features primarily describe the local content and structural information of an image, capturing representative objects, scenes, or other important visual elements. By extracting these region features, we can better understand the specific details in the image and the relationships between its various parts.

[0025] The decoder contains a multi-scale deformable attention module and a self-attention module.

[0026] The decoder portion of Defornable DETR is used to extract the region features and predict the coordinates of the detection box, specifically: Multi-scale feature maps extracted by neural network modules ,in, representing the first layer of feature maps, having different spatial resolutions and channel numbers; initial feature vectors learned where each vector is the first feature vector, is the total number of feature vectors; In the attention module of the decoder, the object query interacts with each vector and the convolution feature map , specifically: The convolution feature map is the key and value, and each vector is the query, which interacts through a multi-scale variability attention mechanism. The decoder is composed of six layers of multi-scale variability attention modules, and the output is where, is the object query feature after the attention mechanism is updated for each layer; is the object query feature after the attention mechanism is updated for each layer; at the last layer of the decoder, the output is the final region feature, represented as: (4); where, is the first region feature vector, is the number of region features extracted by DETR; is the dimension of each region feature vector; The region coordinates are obtained through linear layer processing, and the format is , which is the geometric feature corresponding to the region feature.

[0027] Referring to Figure 2 , the structured semantic feature is constructed, specifically: Given an input image , the image is divided into subgraph sets, and each subgraph represents a local region of the image; For the entire image, cross-modal retrieval is performed using the CLIP model to obtain words that are most relevant to the global semantics of the image, where each word corresponds to a text feature vector; At the subgraph level, independent cross-modal retrieval is performed on each subgraph using the CLIP model to obtain a word , wherein is the number of words related to the th subgraph; and the semantic information of the whole graph is gradually compensated through this process until non-repeated words are accumulated from all subgraphs; For the obtained word set , the K-Means clustering algorithm is used to divide the words into clusters, obtaining a first layer of prototype sets , wherein is the number of prototypes representing a group of semantically similar words, ; each prototype represents a group of semantically similar words; then the first layer of prototype sets is further clustered to obtain a second layer of prototype sets , wherein is the number of clusters in the second layer, and each prototype represents a group of semantically broader concepts, and the operation is performed in a loop to obtain higher-level prototype sets layer by layer until the final hierarchical semantic representation of the image is constructed, and the expression is: (5); wherein Cluster(X,K) represents K-Means clustering of the set to generate clusters, is the number of clusters in the th layer; the obtained prototypes of each layer represent the multi-level semantic information of the image, and the high-level prototypes represent the main body or category information of the image; the low-level prototypes represent the specific details or local features of the image.

[0028] Through the hierarchical clustering strategy, the fine-grained semantic information of the image can be captured from multiple levels, not only improving the expression ability of local features, but also enhancing the integration of global semantic structure, thereby improving the accuracy, coherence and semantic consistency of the generated captions.

[0029] Referring to Figure 3 , the specific process of the geometric perception semantic reinforcement encoder is as follows: To extract the position information, the absolute position features and the relative position features of the region geometric features and the grid geometric features need to be calculated first. For the object region, the absolute position encoding is: (6); where Norm denotes the normalization, and denote the top-left and bottom-right corners of the bounding box, respectively; is the size of the whole image; is an embedding parameter matrix; The absolute position information of the grid features is different from that of the region features. Since each image has fixed coordinate information, the trigonometric functions are used to encode in a fixed way and establish the absolute position relationship of the network features using one-dimensional sine and cosine embedding, which is expressed as: (7); (8); where is the number of the image grid in the row or column direction; when is even, the position information is encoded using the sine function; when is odd, the cosine function is used for encoding; is The constant 10000 is used to regulate the change rate of the sine and cosine functions at different frequencies, and different dimensions of encoding use different frequencies; The absolute position encoding of the grid is: G A B ( i , j ) = [ E i , E j ] (9); where and are the row index and column index of the grid, respectively, to obtain the absolute position feature of the region and the absolute position feature of the grid ; and denote the absolute position encoding of the image grid in the row direction and the column direction, respectively; For the relative position feature, the representation methods of the region and the grid are the same, because the grid can be regarded as a special bounding box.

[0030] For any two bounding boxes and , the relative position vector is calculated using the geometric coordinate information , and the formula is: λ = [ l o g ( | c x m − c x n | w m ) , l o g ( | c y m − c y n | h m ) , l o g ( w n w m ) , l o g ( h n h m ) , I o U ( m , n ) ] (10); where and are the coordinates of the center point of the bounding box; and are the height and width of the bounding box; is the bounding box and the intersection over union; The relative position encoding is embedded into the high-dimensional encoding, and the formula is: (11); (12); wherein, is the relative position feature of the region feature; is the relative position feature of the network feature.

[0031] The absolute position feature of the region feature has been obtained and the relative position feature of the region feature and the absolute position feature of the network feature and the relative position feature of the network feature After that, the position features of the region and the network need to be fused respectively.

[0032] The specific process of the adaptive weighted feature decoder is as follows: In the adaptive weighted feature decoder, the input sequence represents the language feature in the autoregressive decoding stage; The decoding process produces interaction information through the masked self-attention mechanism , which is used to model the dependency between language features, and the specific formula is: (13); wherein, is the masked multi-head self-attention; is the interaction information; The interaction information is sent to three cross-modal attention modules to interact with the network feature , the region feature and the structured semantic feature to extract multi-modal information; For the first cross-modal attention module, the interaction information serves as the query vector, the network feature serves as the key and value, and the attention weight between the query vector and the region feature is: (14); In the second cross-modal attention module, the interaction information serves as the query vector, the region feature serves as the key and value, and the generated attention weight is: (15); In the third cross-modal attention module, interaction information As a query vector, structured semantic features As keys and values, generated attention weights For: (16); Each set of attention weights generated subsequently is concatenated with the query vector The combined vector is normalized by a Sigmoid function to obtain a probability distribution , , The formula is: c i r = S i g m o i d ( W r [ a i r ; x i * ] + b r ) (17); c i g = S i g m o i d ( W g [ a i g ; x i * ] + b g ) (18); c i s = S i g m o i d ( W s [ a i s ; x i * ] + b s ) (19); Wherein, , , , , , are learnable parameters; the probability distribution , , is multiplied by the corresponding attention weight , , respectively, and the final output feature is calculated by a normalization layer, and the formula is: (20); Wherein, is layer normalization; is element-wise multiplication; is a conversion matrix.

[0033] The above-described embodiments only express the implementation of the present application, and the description is more specific and detailed, but it cannot be understood as limiting the scope of the patent of the present application. It should be noted that for ordinary skilled persons in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the protection scope of the patent of the present application should be based on the appended claims.

Claims

1. An image captioning algorithm based on structured semantic extraction and geometric feature fusion, characterized in that, include: Visual semantic feature extraction module and subtitle generation module; The visual semantic feature extraction module includes: Region features: used to capture local details and key object information in an image; Network features: provide global contextual semantics; Structured semantic features: By introducing CLIP retrieval and multi-level clustering, semantic representations with hierarchical relationships are extracted; The subtitle generation module uses the extracted multimodal features as a basis, and decodes and integrates the features through a deep neural network to transform them into fluent and semantically accurate natural language descriptions.

2. The image captioning algorithm based on structured semantic extraction and geometric feature fusion according to claim 1, characterized in that, The extraction process of region features, network features, and structured semantic features in the visual semantic feature extraction module is as follows: Mesh features and network geometric features: Extraction of mesh category features involves dividing the image into a fixed-size mesh and using a neural network model to extract contextual features from each mesh and coordinate information of each network. Region features and region geometric features: The extraction of region features is achieved through an object detector, which performs region-based analysis on the image and extracts the feature representation of each object, as well as the object's coordinate quadruple; Structured semantic features: The CLIP model is used to perform cross-modal retrieval of images and the Top-K text features most relevant to the images are obtained from the text corpus. The extracted text features are clustered using a multi-level clustering algorithm to construct a hierarchical semantic representation and capture the semantic associations and hierarchical relationships between concepts.

3. The image captioning algorithm based on structured semantic extraction and geometric feature fusion according to claim 2, characterized in that, The subtitle generation module includes: Geometric-aware semantic enhancement encoder: By progressively fusing grid features, region features, and structured semantic features, and introducing geometric coordinate information during the fusion process, the expressive power of structured semantic features is enhanced. Adaptive weighted feature decoder: Utilizing the fused global feature representation, along with grid features and fused region features, the grid features are weighted and fused together, and then generated word-by-word subtitles describing the image content through autoregressive decoding. During the decoding process, the contributions of visual and semantic information are dynamically balanced to ensure that the generated subtitles are semantically coherent and accurately reflect the image content and spatial relationships.

4. The image captioning algorithm based on structured semantic extraction and geometric feature fusion according to claim 1, characterized in that, The neural network model is used as a feature extractor to obtain multi-scale features and global contextual information of the image, specifically: Given an image First, adjust the image size to The size is determined and divided into individual non-overlapping blocks, each block having a resolution of [resolution value missing]. By pre-embedding, each of the small blocks is converted into a feature vector, resulting in the following initial feature sequence: （1）； in, This is the initial feature sequence of image I; The feature vector of the Nth small block; For the first Feature vectors of small blocks; for 3D real space; For feature dimensions; The feature sequence is processed through a hierarchical network of a neural network model to extract multi-scale features. In each layer, sliding window attention and cross-window attention mechanisms are applied to capture local and global relationships and update the feature representation. The formula is as follows: （2）； Simultaneously, at each stage, except for the last layer, the spatial dimension of the feature map is downsampled by a factor of 2 through the Patch Merging operation, thereby generating a multi-scale feature map, represented as: （3）； in, Multi-scale feature maps; Total number of floors; The height of the image; For the first The feature dimensions of the layer; Geometric coordinates are calculated using the resolution of the input image and the size of the multi-scale feature map. ,in, For the first The region's geometric coordinates; use and Geometric features of image regions and regional features.

5. The image captioning algorithm based on structured semantic extraction and geometric feature fusion according to claim 4, characterized in that, The decoder portion of the deformable DETR is used to extract the region features and predict the coordinates of the detection box, specifically: Multi-scale feature maps extracted by neural network modules ,in, Indicates the first The feature maps of the layers have different spatial resolutions and number of channels; Initial feature vectors for learning , where each vector It is the first 1 eigenvector It is the total number of eigenvectors; In the attention module of the decoder, the object queries each vector. and convolutional feature maps They interact with each other, specifically as follows: Convolutional feature maps Each vector serves as both a key and a value. As a query, interaction is achieved through a multi-scale variable attention mechanism. The decoder consists of six layers of multi-scale variable attention modules, and the output is... ,in, To represent the object query features updated by the attention mechanism at each layer; to represent the object query features updated by the attention mechanism at each layer; in the last layer of the decoder, the output is... As the final regional feature, it is represented as: （4）； in, for Indicates the first Each region's feature vector, It is the number of region features extracted by DETR; It is the dimension of the feature vector of each region; Region coordinates are obtained through linear layer processing. The format is , which serves as the geometric feature corresponding to the regional feature.

6. The image captioning algorithm based on structured semantic extraction and geometric feature fusion according to claim 5, characterized in that, The structured semantic features are constructed as follows: Given an input image Divide the image into Sub-sets, each sub-graph Represents a local region of the image; For the entire image, the CLIP model is used for cross-modal retrieval to obtain the most relevant data to the global semantics of the image. one word , where each word Each corresponds to a text feature vector; At the subgraph level, the CLIP model is used for each subgraph. Perform independent cross-modal retrieval to obtain the relevant data for each subgraph. one word ,in, For the first The number of words related to each subgraph is counted; and the semantic information of the entire graph is gradually compensated through this process until the total semantic information is retrieved from all subgraphs. 100 unique words; among them, This represents the total number of unique words across all subgraphs; for the resulting word set, The K-Means clustering algorithm was used to divide these words into groups. This yields a set of clusters, resulting in the first layer of fine-grained prototypes. ,in, For each prototype Represents a group of semantically similar words. Each prototype Represents a group of semantically similar words; followed by the prototype set of the first level. Further clustering yields the prototype set for the second layer. ,in, The number of clusters in the second layer, and each prototype Representing a set of semantically broader concepts, it operates through iterations, progressively clustering to obtain higher-level prototype sets. This continues until the final hierarchical semantic representation of the image is constructed, expressed as: （5）； Where Cluster(X,K) represents performing K-Means clustering on the set to generate Clusters, For the first The number of clusters in each layer; the resulting prototypes of each layer This represents the multi-layered semantic information of an image. Higher-level prototypes represent the image's subject or category information; lower-level prototypes represent... This indicates the specific details or local features of the image.

7. The image captioning algorithm based on structured semantic extraction and geometric feature fusion according to claim 3, characterized in that, The specific process of the geometric-aware semantic enhancement encoder is as follows: To extract location information, it is first necessary to analyze the geometric features of the region. and mesh geometry features Calculate the absolute position features and relative position features separately. For the object region, the absolute position encoding is as follows: （6）； Where Norm represents normalization. and These represent the top left and bottom right corners of the bounding box, respectively. The size of the entire image; It is an embedding parameter matrix; The absolute positional information of grid features differs from that of region features. Since each image has fixed coordinate information, trigonometric functions are used to encode the information in a fixed manner, and one-dimensional sine and cosine embeddings are used to establish the absolute positional relationship of the network features, represented as: （7）； （8）； in, Numbering of the image grid in the row or column direction; when When the number is even, a sine function is used to encode the position information; when... When the number is odd, cosine function encoding is used; for The constant 10000 is used to control the rate of change of the sine and cosine functions at different frequencies, and different frequencies are used for encoding in different dimensions; The absolute position encoding of the grid is: （9）； in, and The row and column indices of the grid are used to obtain the absolute location features of the region. and the absolute position features of the grid ; and These represent the absolute positional encoding of the image grid in the row and column directions, respectively; For any two bounding boxes and Using geometric coordinate information Calculate the relative position vector The formula is: （10）； in, and These are the coordinates of the center point of the bounding box; and The height and width of the bounding box; bounding box and The intersection and union ratio; The relative position encoding is embedded into the high-dimensional encoding, as shown in the formula: （11）； （12）； in, The relative positional features of the region; This refers to the relative positional features of the network features.

8. The image captioning algorithm based on structured semantic extraction and geometric feature fusion according to claim 3, characterized in that, The specific process of the adaptive weighted feature decoder is as follows: In the adaptive weighted feature decoder, the input sequence Represents linguistic features in the autoregressive decoding stage; The decoding process generates interactive information through a masked self-attention mechanism. It is used to model the dependencies between language features, specifically: （13）； in, For masking multi-head self-attention; For interactive information; Interactive information They are fed into three cross-modal attention modules, which are respectively connected to network features. Regional characteristics and structured semantic features Interact with the system to extract multimodal information; For the first cross-modal attention module, the interaction information Network features as query vectors Attention weights between the query vector and region features, serving as both keys and values. for: （14）； In the second cross-modal attention module, interactive information As a query vector, region features The generated attention weights serve as keys and values. The formula is: （15）； In the third cross-modal attention module, interactive information Structured semantic features as query vectors The generated attention weights serve as keys and values. for: （16）； Each subsequently generated attention weight and query vector The concatenated vectors are combined and normalized using the Sigmoid function to obtain the probability distribution. , , The formula is: （17）； （18）； （19）； in, , , , , , It is a learnable parameter; probability distribution , , Each with its corresponding attention weight , , Multiply the components and calculate the final output features through a normalization layer, as shown in the formula: （20）； in, For layer normalization; For element-wise multiplication; This is the transformation matrix.

Images