News graph set description generation method and system based on graph clustering and multi-graph text interaction

Abstract

The application provides a news album description generation method and system based on graph clustering and multi-graph text interaction, which comprises data acquisition and data labeling, collecting news album data from a news album website, removing invalid information in the text, and storing the corrected text in a data set; data preprocessing, normalizing the images in the generated data set; performing named entity recognition on the text in the generated data set and generating a corresponding entity word list; graph feature extraction and exploration of the implicit relationship between album images, the graph clustering module explores the implicit relationship between album images, performs principal component analysis dimension reduction and clustering analysis on the extracted image coarse-grained features, mines the structured information between album images, obtains image classification clusters and multiple cluster centers; graph-text information interaction, model training, and using an end-to-end method to train the model, and adopting a cross-entropy loss function. The application supports description generation for news albums containing external text information.

Description

Technical Field

[0001] This invention belongs to the field of image description generation, which is an intersection of natural language processing and computer vision. In particular, it relates to a method for generating news photo atlas descriptions based on graph clustering and multi-image-text interaction. Specifically, it uses a graph clustering module to explore the implicit relationships between images in the atlas for image feature optimization. Then, it combines the graph clustering module with ITBGA to complete the information interaction between multiple images, structured information between images, and text. Finally, it uses pointer networks to improve the final description text generation method. Background Technology

[0002] With the development of the internet and information technology, multimedia data is experiencing explosive growth. The amount of data, including images, videos, and text, is increasing, making it crucial to utilize artificial intelligence to quickly and effectively learn from and extract valuable information from this massive dataset. Currently, data exists in increasingly complex forms, with different modalities often combining to convey information; therefore, multimodal data is gradually becoming a research hotspot. In recent years, computer vision and natural language processing have both developed rapidly in their respective fields, driving research into multimodal tasks at their intersection. Image description generation is a classic multimodal task, aiming to generate text descriptions for an image. It has significant practical value in daily life, such as assisting blind readers and automatically generating captions for images.

[0003] The ultimate goal of image description generation is to generate descriptions that fully express the logical semantic relationships between visual objects within a specific context. However, existing methods tend to generate descriptive texts that focus on general objects (such as "a cat" or "a person"), lacking connection to real-world situations (such as place names, personal names, and building names). Therefore, traditional image description generation tasks still have limitations in practical applications, especially in the context of multimedia data overload. General image descriptions lack specific scene meaning and cannot meet the actual needs of current production and daily life.

[0004] News image description generation tasks have addressed the aforementioned lack of relevance to real-world contexts to some extent. This task uses news data as its research object, aiming to generate descriptive text for news images using the contextual information provided by news articles, involving multimodal information interaction between images and text. Since news itself originates from the real world, articles often contain many important named entities, which can assist the model in generating scene-specific descriptive text, enhancing its relevance to real-world contexts. However, news image description generation tasks neglect another important characteristic of news data: news often contains multiple images, and there is structured information between these images. Existing news image description generation tasks only target individual images, not the entire image set, thus compromising its integrity; furthermore, images with similar content often have similar or even identical individual descriptions, leading to redundancy. Therefore, describing news image sets as a whole is more reasonable. In recent years, there have been some studies on summarizing image sets, but most do not involve textual descriptions, only summarizing by selecting smaller subsets from the image set. Consequently, existing methods lack exploration of the implicit relationships between images in an image set. Introducing clustering algorithms into the research on descriptive generation of news image atlases can extract structured information between images from the atlas itself, making up for the current lack of exploration of implicit relationships between images in the atlas. This is necessary and reasonable, and has important research significance and value for the descriptive generation of news image atlases. Current research on multimodal information interaction between text and images, such as "A News Event Search Method and System Based on a Multi-Level Text-Image Semantic Alignment Model," only completes the association of text-image features by mapping them to the same feature space, lacking exploration of the inherent attribute connections between the text-image features themselves; "A Visual Question Answering Task Processing Method, System, Electronic Device, and Storage Medium" encodes text-image information independently and directly obtains text-image fusion features through concatenation, lacking deep interaction at the abstract level between text-image features. In this invention, the ITBGA module can fully utilize the input multimodal information to focus on the bidirectional deep interaction of text and image information from both coarse and fine granular levels. Moreover, its module structure supports feature interaction of multiple text and image information, which is more in line with the needs of news photo collection tasks. At the same time, the characteristics of the attention mechanism allow the model to selectively focus on important information, and the attention weight matrix can also play a significant role in the generation module.

[0005] Furthermore, current research on news data clustering analysis, such as "A Hybrid Granularity Multi-View News Data Clustering Method," uses the text descriptions corresponding to news texts, images, and videos as clustering objects, essentially clustering text features. In fact, while text descriptions corresponding to images and videos can summarize the content to some extent, they discard many important details. This invention, however, directly clusters the image set itself, making fuller use of detailed information, accurately controlling the self-division of the image set, and mapping the division results to abstract relationships between images in the image set, serving as new information input for further interaction of text and image features. With the development of the internet and information technology, multimedia data is experiencing explosive growth. With increasingly more image, video, and text data, how to quickly and effectively learn from and extract valuable information from massive amounts of data using artificial intelligence technology has become an important research topic. Currently, the forms of data are becoming increasingly complex, and data of different modalities often convey information through combinations, thus multimodal data is gradually becoming a research hotspot. In recent years, computer vision and natural language processing have developed rapidly in their respective fields, also driving research on multimodal tasks in their intersection. Image description generation is a classic multimodal task that aims to generate text descriptions for an image. It has great practical value in daily life, such as assisting blind people in reading and automatically generating captions for pictures.

[0006] The ultimate goal of image description generation is to generate descriptions that fully express the logical semantic relationships between visual objects within a specific context. However, existing methods tend to generate descriptive texts that focus on general objects (such as "a cat" or "a person"), lacking connection to real-world situations (such as place names, personal names, and building names). Therefore, traditional image description generation tasks still have limitations in practical applications, especially in the context of multimedia data overload. General image descriptions lack specific scene meaning and cannot meet the actual needs of current production and daily life.

[0007] The news image description generation task addresses, to some extent, the problem of lack of relevance to real-world contexts mentioned above. This task uses news data as its research object, aiming to generate descriptive text for news images using the contextual information provided by news articles. Since news itself originates from the real world, it often contains many important named entities, which can assist the model in generating scene-specific descriptive text, enhancing its relevance to real-world situations.

[0008] However, news image description generation tasks overlook another important characteristic of news data: news often contains multiple images, and there is structured information between these images. Existing news image description generation tasks only focus on individual images, failing to consider the image set as a whole, thus compromising its integrity; furthermore, images with similar content often have similar or even identical individual descriptions, leading to redundancy. Therefore, describing news image sets as a whole is more reasonable. In recent years, there have been some studies on summarizing image sets, but most do not involve textual descriptions, only summarizing by selecting smaller subsets from the image set. Consequently, existing methods lack exploration of the implicit relationships between images in an image set. Introducing clustering algorithms into news image set description generation research can extract structured information between images from the image set itself, addressing the current lack of exploration of implicit relationships between images in an image set. This is necessary and reasonable, and has significant research value and importance for news image set description generation. Summary of the Invention

[0009] News photo atlases contain news text information, structured information between images, and image content information. However, due to task differences, existing image description generation methods lack exploration of the structured information between images in the atlas, and existing image-text interaction mechanisms struggle to achieve information interaction among these three elements. To address the shortcomings of the existing technologies, this invention provides a method capable of autonomously exploring the structured information between images in an atlas, thereby enabling information interaction among these three elements (multiple images, structured information between images, and text information) for news photo atlas description generation.

[0010] This invention provides a method for generating news image descriptions based on graph clustering and multi-image-text interaction, comprising the following steps:

[0011] Step 1, Data Collection and Labeling: Collect news photo data from news photo websites, remove invalid information from the text, label and proofread, and then store it in the dataset;

[0012] Step 2, data preprocessing: normalize the images in the generated dataset; perform named entity recognition on the text in the generated dataset and generate a list of corresponding entity words;

[0013] Step 3: Image and text feature extraction and exploration of implicit relationships between images in the atlas. The text in the preprocessed dataset is input into BERT to obtain text feature encoding, and the bag-of-words model is used to encode entity words. The images in the preprocessed dataset are input into the deep residual model to obtain image feature encoding. The graph clustering module explores the implicit relationships between images in the atlas. Principal component analysis is used to reduce the dimensionality of the extracted coarse-grained features of the images, and cluster analysis is performed to mine the structured information between the images in the atlas, and obtain image classification clusters and multiple cluster centers.

[0014] Step 4, Image-Text Information Interaction: The graph clustering module is combined with the multi-image-text interaction ITBGA module. Text feature encoding, image feature encoding, and structured information between images in the image set are used as inputs, supplemented by image classification cluster information and cluster centers, to complete the multi-image-text information interaction process. The decoder is used to decode the interaction results, and the weight distribution generated during the interaction process is input into the pointer network to generate the final descriptive text.

[0015] Step 5, Model Training: The model is trained in an end-to-end manner, using the cross-entropy loss function. The trained model is then used to support the generation of descriptions for news image sets containing external text information.

[0016] Furthermore, for the news text portion, named entities are extracted, and the overall text features T are obtained using the pre-trained model BERT, denoted as T = {t1, t2, ..., t}. L}, where L is the length of the text. Let d represent the text features of the i-th word, with the feature vector having dimension d. t For named entities, the encoding vector of the i-th word is obtained through the Word2Vec bag-of-words model. and position encoding vector Both have a dimension of d. t Then, the final named entity feature representation E = {e1, e2, ..., e} is obtained through formula (1). K}, where K represents the number of named entities:

[0017] e i =w i +p i (1)

[0018] Furthermore, for an image set V containing M images, ResNet152 is used to extract coarse-grained and fine-grained visual features of V, respectively, denoted as V C ={v1,v2,…,v M},in Let represent the coarse-grained visual features of the i-th image in the image set, with the feature vector having dimension d. v ; in This represents the fine-grained visual features of the i-th image in the image set, where N is the number of image regions within an image. Let represent the fine-grained visual features of the j-th region within the image, with the feature vector having dimension d. v .

[0019] Furthermore, before engaging in image-text interaction, the graph clustering module is used to perform PCA dimensionality reduction and clustering on the image features to obtain new image feature representations.

[0020] Obtained through clustering The specific process is as follows:

[0021] U,S=SVD(V C (2)

[0022] U′=U(:,0:512) (3)

[0023] X = V C ·(V C -S) T ·U′ (4)

[0024] clu ids ,clu cen =kmeans(X) (5)

[0025] Num clu =count(clu) ids (6)

[0026] P clu =f P (Num clu (7)

[0027]

[0028] Wherein, SVD represents performing singular value decomposition to obtain the eigenvector matrix U and the square root of the eigenvalue S; formula (3) represents truncating U in the second dimension, retaining only the first 512 dimensions; X represents the image features after PCA dimensionality reduction; kmeans represents performing K-Means clustering, pre-specifying the number of clusters, and obtaining the cluster category ID list clu to which each image belongs. ids And the cluster center vector representation of each cluster (clu) cen ;count represents the list of images in each cluster, calculated based on the cluster category list corresponding to each image;f P This represents transforming the distribution of the number of images within each cluster into the corresponding weight distribution P. clu The specific process is as follows:

[0029] P tmp =softmax(Num clu (9)

[0030]

[0031]

[0032] Among them, P tmp A list of weights corresponding to each cluster; fP The corresponding weight distribution can be directly derived from the distribution of the number of images, that is, the larger the number of images in a cluster, the greater the weight ratio.

[0033] Furthermore, the implementation of step 4 includes the following sub-steps:

[0034] 1) Inputting image and text features into the first part of the clustering-integrated ITBGA module, namely the clustering-integrated CITBGA module, firstly, the feature vectors undergo simple processing and transformation. Then, using two MHAoA structures, image and text features are used as query vectors Q respectively, and they interact with each other as K and V information, to obtain the final feature representation.

[0035] T′=selfAtt(T) (12)

[0036]

[0037]

[0038] Where selfAtt represents the self-attention layer, and [;] represents the vector concatenation operation;

[0039] 2) Input the image and text features output in step 1), and the clustered image weight distribution, into the second part of the clustering combined ITBGA module, namely the enc-FITBGA module for clustering combined.

[0040]

[0041]

[0042]

[0043] T v =MHAoA(T V ,v,v) (18)

[0044] T v ′=LayerNorm(T′+T v (19)

[0045] T enc =LayerNorm(T v ′+FFN(T v ′)) (20)

[0046] Where Avgpool represents T V Average pooling is performed to obtain the overall feature representation of the text; A T Therefore As a query, with V TAs the key, the weight matrix is ​​obtained after multi-head attention (MHAtt); choice represents the image selection operation, which determines the image with the highest attention, and the feature vector is represented as v; LayerNorm represents the input normalization operation; FFN represents a two-layer feedforward network with ReLU as the activation function after the fully connected operation;

[0047] Subsequently, the CITBGA and enc-FITBGA processes will be repeated for named entity feature E and V. T The interaction is performed, and the final overall visual features are represented as V. enc Named entity features are represented as E enc ;

[0048] 3) Input the image and text features output in step 2), the clustered image weight distribution, and the graph cluster centers into the third part of the clustering combined ITBGA module, namely the clustering combined dec-FITBGA module.

[0049]

[0050]

[0051]

[0052]

[0053]

[0054]

[0055]

[0056]

[0057]

[0058] Among them, att T ,att E They represent Two different attention weight matrices are generated after the background text and named entities interact with each other through the MHAoA module; clu represents the cluster center cen The average vector; a clu This represents the attention weights related to entity words obtained after the interaction, which will be used in the pointer network to influence entity word generation. To obtain richer weight information, visual features v′ are used as the query and T. enc Perform multi-head attention interaction to obtain the weight distribution.

[0059]

[0060] Output The probability distribution is transformed using softmax and then used for word generation.

[0061]

[0062] 4) The result obtained in step 3) att T ,att E Averaging The weight distributions a of background text words and named entity words during the generation process were obtained respectively. T and a E , where a T It includes information on fine-grained text-image interactions; then, the two are mapped onto the generated vocabulary to obtain the weight distribution P on the vocabulary. T and P E These are the generation probability distributions of background text words and named entity words, respectively.

[0063]

[0064]

[0065] P T =f(a T (34)

[0066] P E =f(a E (35) where f represents the mapping process; the decoding vector of the dec-FITBGA module The probability values ​​p and q are calculated, representing the probability of generating a word from the background text and the probability of generating a word from the named entity list, respectively; the calculation process is as follows.

[0067]

[0068]

[0069] Among them, W p and W q These are learnable parameters, and σ is the sigmoid function; the final probability distribution of the predicted text is... The calculation process is as follows:

[0070]

[0071] Furthermore, the NISC model is trained using the cross-entropy loss function based on the probability distribution values ​​obtained in step 4.

[0072]

[0073] Among them, P * This represents the probability distribution of the true labels.

[0074] On the other hand, the present invention provides a news image set description generation system based on graph clustering and multi-image-text interaction, which is used to implement the news image set description generation method based on graph clustering and multi-image-text interaction as described above.

[0075] Moreover, it includes the following modules,

[0076] The first module is used for data collection and data annotation. It collects news photo data from news photo websites, removes invalid information from the text, and stores the data into the dataset after annotation and proofreading.

[0077] The second module is used for data preprocessing, which normalizes the images in the generated dataset and performs named entity recognition on the text in the generated dataset to generate a list of corresponding entity words.

[0078] The third module is used for image and text feature extraction and exploration of implicit relationships between images in the atlas. The text in the preprocessed dataset is input into BERT to obtain text feature encoding, and the bag-of-words model is used to encode entity words. The images in the preprocessed dataset are input into the deep residual model to obtain image feature encoding. The graph clustering module explores the implicit relationships between images in the atlas. Principal component analysis is used to reduce the dimensionality of the extracted coarse-grained features of the images and cluster analysis is performed to mine the structured information between the images in the atlas and obtain image classification clusters and multiple cluster centers.

[0079] The fourth module is used for image-text information interaction. It combines the image clustering module with the multi-image-text interaction ITBGA module. It takes text feature encoding, image feature encoding, and structured information between images in the image set as input, supplemented by image classification cluster information and cluster centers, to complete the multi-image-text information interaction process. The decoder is used to decode the interaction results, and the weight distribution generated during the interaction process is input into the pointer network to generate the final descriptive text.

[0080] The fifth module is used for model training. It uses an end-to-end approach to train the model, employs the cross-entropy loss function, and utilizes the trained model to support the generation of descriptions for news image sets containing external text information.

[0081] Alternatively, it may include a processor and a memory, with the memory used to store program instructions and the processor used to call the stored instructions in the memory to execute a news atlas description generation method based on graph clustering and multi-graph-text interaction as described above.

[0082] Alternatively, it may include a readable storage medium storing a computer program that, when executed, implements a news atlas description generation method based on graph clustering and multi-image-text interaction as described above.

[0083] This invention supports the generation of descriptions for news photo sets containing external text information, solving the problems of existing methods lacking exploration of the relationships between images in the photo set and difficulty in completing multi-image and text information interaction. It improves the practicality of related technologies and further mines the structured information of data, thus having certain market value. Attached Figure Description

[0084] Figure 1 This is a flowchart of an embodiment of the present invention;

[0085] Figure 2 This is a graph clustering combined with CITBGA module diagram according to an embodiment of the present invention;

[0086] Figure 3 This is a graph clustering combined with enc-FITBGA module diagram according to an embodiment of the present invention;

[0087] Figure 4 This is a graph clustering combined with dec-FITBGA module diagram according to an embodiment of the present invention;

[0088] Figure 5 This is a schematic diagram of a dataset example and annotation tool according to an embodiment of the present invention;

[0089] Figure 6 This is a flowchart illustrating the training process of the graph clustering combined model according to an embodiment of the present invention. Detailed Implementation

[0090] To more clearly and intuitively demonstrate the purpose, technical solution, and advantages of this invention, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0091] This invention discloses a method for generating news image atlas descriptions based on graph clustering and multi-image-text interaction. The technical solution includes: collecting data for news image atlas description generation; building an annotation tool to generate image atlas description pairs based on news content annotations, cleaning and organizing the data to form a news image atlas description generation dataset; model training, including training the NISC model using the dataset from step 2; data preprocessing, processing news images to a uniform size, and performing named entity recognition on the news text; feature extraction, using an encoder to convert images, text, and named entities into corresponding feature vector representations; utilizing a graph clustering module to perform principal component analysis for dimensionality reduction and cluster analysis on coarse-grained image features to mine structured information between image atlas images, obtaining image classification clusters and multiple cluster centers; combining the graph clustering module with the ITBGA module to complete information interaction between multiple images, structured information between images, and text; and using a decoder to decode the output features and generate pointer network auxiliary words to obtain the final news image atlas description text.

[0092] First, the technical terminology of this invention will be explained and described:

[0093] BERT: BERT stands for Bidirectional Encoder Representations from Transformers, a novel language model developed and released by Google in late 2018. Traditional neural machine translation mostly uses RNNs or CNNs as encoding and decoding model libraries, while BERT abandons the traditional RNN and CNN models. It uses an attention mechanism to convert the distance between two words at any position into 1, effectively solving the thorny long-term dependency problem in natural language processing. As a pre-trained language model, BERT essentially learns a good feature representation for words through self-supervised learning on massive corpora. Users can directly use BERT as the Word2Vec transformation matrix and efficiently apply it to their respective tasks, fine-tuning it or fixing it as a feature extractor depending on the task.

[0094] ResNet: ResNet stands for Residual Network, a convolutional neural network proposed by four researchers from Microsoft Research. ResNet is characterized by its ease of optimization and the ability to improve accuracy by significantly increasing its depth. Its internal residual blocks use skip connections to mitigate the vanishing gradient problem that arises when increasing depth in deep neural networks.

[0095] ITBGA: ITBGA stands for Image and Text Bidirectional Guidance Attention.

[0096] CITBGA: CITBGA stands for Coarse-grained Image Text Bidirectional GuidanceAttention, which is a form of ITBGA, namely coarse-grained image and text bidirectional guidance attention.

[0097] FITBGA: FITBGA stands for Fine-grained Image Text Bidirectional GuidanceAttention, a form of ITBGA that provides fine-grained bidirectional guidance for image and text attention. enc-FITBGA is located at the encoding layer, while dec-FITBGA is located at the decoding layer. Their structures are similar but not identical, and they are combined with the clustering module in different ways.

[0098] Softmax: The Softmax function is the most widely used activation function. It transforms the model's prediction results into an exponential function and then normalizes the transformed results.

[0099] This invention discloses a method for generating news image atlas descriptions based on graph clustering and multi-image-text interaction. The technical solution includes: collecting data for news image atlas description generation; building an annotation tool to generate image atlas description pairs based on news content annotations, cleaning and organizing them to form a news image atlas description generation dataset; model training, including training an NISC model using the dataset; data preprocessing, processing news images to a uniform size, and performing named entity recognition on news text; feature extraction, using an encoder to convert images, text, and named entities into corresponding feature vector representations; utilizing a graph clustering module to perform principal component analysis for dimensionality reduction and cluster analysis on coarse-grained image features to mine structured information between image atlas images, obtaining image classification clusters and multiple cluster centers; combining the graph clustering module with an ITBGA module to complete information interaction between multiple images, structured information between images, and text; and using a decoder to decode the output features and generate pointer network auxiliary words to obtain the final news image atlas description text.

[0100] See Figure 1 This invention provides a method for generating news image descriptions based on graph clustering and multi-image-text interaction, comprising the following steps:

[0101] Step 1: First, construct the news image atlas description generation dataset:

[0102] Step 1.1: Use web scraping technology to collect 10,000 news photo collections from multiple news photo collection websites. After adding basic restrictions, use code to perform preliminary cleaning to remove illegal characters, useless spaces and other invalid information from the text.

[0103] Step 1.2: Use the Vue front-end framework to build a visual annotation platform, write the description text according to the annotation specifications, and then have it checked by the team leader before storing it in the dataset.

[0104] The example collected 10,000 news photo albums from Beijing News Pictorial (https: / / www.bjnews.com.cn), Sina Pictures (http: / / photo.news.sina.com.cn), and China News Photo Network (http: / / www.cnsphoto.com). After adding basic restrictions, the code was used to perform preliminary cleaning to remove illegal characters, useless spaces, and other invalid information from the text.

[0105] In practice, images within a single dataset need to be filtered to maintain a number between 3 and 8. Then, based on the news background text and the image content, descriptive text for the image atlas should be written and annotated, summarizing the content of all images as comprehensively as possible based on the information provided by the news background text. Dataset examples and annotation tools are as follows: Figure 5 As shown. Alternatively, pre-screening and labeling can be performed, and the resulting dataset can be used as input to directly proceed to step 2.

[0106] Step 2: Perform further data preprocessing on the labeled dataset.

[0107] Step 2.1: First, preprocess the images by reducing their size to 256*256. Second, since the number of images in each data set is uncertain, after determining the maximum number of images in a single data set, set the maximum number of images to 8; if the number is insufficient, padding with a matrix of all zeros.

[0108] Step 2.2: News articles are divided into two parts: title and body. First, the title and body are concatenated. Second, on the one hand, the maximum text length is set to 512, with insufficient text padded with 0s and excess text discarded; on the other hand, Baidu LAC tool is used for named entity recognition.

[0109] For the news text portion, the preferred embodiment uses the Baidu LAC tool to extract named entities and utilizes the pre-trained BERT model to obtain the overall text features T, denoted as T = {t1, t2, ..., t}. L ], where L is the length of the text, Let d represent the text features of the i-th word, with the feature vector having dimension d. t For named entities, the implementation first obtains the encoding vector of the i-th word using the Word2Vec bag-of-words model. and position encoding vector Both have a dimension of d. tR is the real number field, and the final named entity feature representation E = {e1, e2, ..., e} is obtained through formula (1). K}, where K represents the number of named entities:

[0110] e i =w i +p i (1)

[0111] Step 3: Extract textual and image features from the organized and preprocessed dataset and explore the implicit relationships between images in the atlas.

[0112] In this embodiment, the news image dataset uses a pre-trained model for feature extraction, which includes the following steps:

[0113] Step 3.1: Text feature extraction. Input the text in the preprocessed dataset into BERT to obtain text feature encoding. For entity words, use the Word2Vec bag-of-words model to encode them and add them to the position encoding to obtain the final encoding.

[0114] Step 3.2: Image feature extraction and structured information extraction between images in the atlas. The images in the preprocessed dataset are input into a deep residual model (preferably a ResNet model in this embodiment) to obtain image feature encoding. Principal component analysis is used to reduce the dimensionality of the coarse-grained features of the images, and cluster analysis is performed to extract the structured information between the images in the atlas, resulting in image classification clusters and multiple cluster centers.

[0115] For an image set V containing M images, this embodiment uses ResNet152 to extract coarse-grained and fine-grained visual features of V, denoted as V0. C ={v1,v2,…,v M},in Let represent the coarse-grained visual features of the i-th image in the image set, with the feature vector having dimension d. v ; in This represents the fine-grained visual features of the i-th image in the image set, where N is the number of image regions within an image. Let represent the fine-grained visual features of the j-th region within the image, with the feature vector having dimension d. v .

[0116] Step 3.3: Before performing image-text interaction, the image features need to be further processed using the graph clustering module through PCA dimensionality reduction and clustering to obtain a new image feature representation. Obtained through clustering The specific process is as follows:

[0117] U,S=SVD(VC (2)

[0118] U′=U(:,0:512) (3)

[0119] X = V C ·(V C -S) T ·U′ (4)

[0120] clu ids ,clu cen =kmeans(X) (5)

[0121] Num clu =count(clu) ids (6)

[0122] P clu =f P (Num clu (7)

[0123]

[0124] SVD() represents performing singular value decomposition to obtain the eigenvector matrix U and the square roots S of the eigenvalues. Formula (3) represents truncating U in the second dimension, retaining only the first 512 dimensions, and the result is denoted as U′. X represents the image features after PCA dimensionality reduction. kmeans() represents performing K-Means clustering, with the number of clusters pre-specified as 3, to obtain the cluster category ID list clu to which each image belongs. ids And the cluster center vector representation of the three clusters clu cen The `count()` function represents a list of cluster categories corresponding to each image. ids The number of images Num in each cluster was statistically obtained. clu A list. f P () indicates that the distribution of the number of images within each cluster is transformed into the corresponding weight distribution P. clu The specific process is as follows:

[0125] P tmp =softmax(Num clu (9)

[0126]

[0127]

[0128] Among them, P tmp This is a list of weights corresponding to the three clusters. This indicates that the weights are obtained based on the cluster category of the i-th image. f represents the weight value corresponding to the i-th image, where i represents the image index. P The corresponding weight distribution can be directly derived from the distribution of the number of images, that is, the larger the number of images in a cluster, the greater the weight ratio.

[0129] Step 4: Use ITBGA (Integrated Truncation and Data Interaction) with clustering to perform image-text information interaction and generate descriptive text. The steps are as follows:

[0130] Cluster-based combined ITBGA includes the cluster-based combined CITBGA module (see [link]). Figure 2 ), clustering combined with the enc-FITBGA module (see Figure 3 ), clustering combined with dec-FITBGA (see Figure 4 The three parts are text feature encoding, image feature encoding, and structured information between images in the atlas, which are input into the cluster-integrated ITBGA, supplemented by image classification cluster information and cluster centers, to complete the multi-image and text information interaction process and obtain the final feature vector encoding and attention weight matrix.

[0131] The final feature vector is decoded using a decoder, input into a fully connected layer and processed by softmax to obtain the predicted probability distribution. At the same time, the attention weight matrix generated during the interaction is input into the pointer network to generate the guide word.

[0132] The specific implementation of this embodiment includes the following sub-steps:

[0133] Step 4.1, Figure 2 The first part of the clustering combined with ITBGA module, namely the clustering combined with CITBGA module, has the following specific structure and process: After inputting image and text features, step 3.3 first performs PCA dimensionality reduction and clustering analysis on the image, and then calculates the weight distribution P obtained after clustering. clu Acting on V C New image feature representations are obtained. right The text features are concatenated, and then two multi-head attention on attention (MHAoA) structures are used. The image and text features are used as query vectors Q, and the concatenated vector is used as K and V information to perform cross-attention interaction.

[0134]

[0135] T′=selfAtt(T) (13)

[0136]

[0137]

[0138] Where selfAtt() represents the self-attention layer, [;] represents the vector concatenation operation, and T V V represents the result of applying MHAoA to T′ and the concatenated vector. T express The result of applying MHAoA to the concatenated vector.

[0139] Step 4.2, Figure 3 This is the second part of the clustering combined ITBGA module, namely the clustering combined enc-FITBGA module. The specific structure and flow are as follows: The T output in step 4.1... V Perform Avgpooling (average pooling) and then combine it with the V output from step 4.1. T Perform multi-head attention interaction and utilize the weight A after the interaction. T Compared with the weight distribution P obtained in step 3.3 clu Combining these methods, the image v with the highest attention is selected through the choice operation, and then T is used... V The feature vector is obtained by performing multi-head attention interaction (MHAoA) with v, and finally by regularization. The specific formula is as follows:

[0140]

[0141]

[0142]

[0143] T v =MHAoA(T V ,v,v) (19)

[0144] T v ′=LayerNorm(T′+T v (20)

[0145] T enc =LayerNorm(T v ′+FFN(T v (21) where Avgpool() is used to process T V Average pooling is performed to obtain the overall feature representation of the text. A T Therefore As a query, with V TThe weight matrix is ​​obtained as the key value after multi-head attention (MHAtt). `choice` represents the image selection operation, determining the image with the highest attention; the selected feature vector is represented as `v`. `LayerNorm()` represents the input normalization operation. `FFN()` represents a two-layer feed-forward network, using ReLU as the activation function after the fully connected operation. T v ' represents the result after text feature regularization, T enc This represents the text feature representation of the final output of the enc-FITBGA module, where v represents the image with the highest attention selected through the choice operation.

[0146] Subsequently, steps 4.1 and 4.2 will be repeated for named entity feature E and V. T The interaction is performed, and the final overall visual features are represented as V. enc Named entity features are represented as E enc .

[0147] Step 4.3, Figure 4 The third part of the clustering combined with ITBGA module, namely the clustering combined with dec-FITBGA module, has the following specific structure and process: First, the label text is represented by an embedding layer as X = {x0, x1, ..., x...} n}, where n represents the length of the tag text. Let d represent the text features of the i-th word, with the feature vector having dimension d. t Then, through an MHAoA layer with a mask, the text x at time t is made... t Query the text before time t, retain key information, and obtain the vector after attention. Then compare it with the image features V output in step 4.2. enc Performing an HMAt interaction will generate attention weight A. x Using A x Compared with the weight distribution P generated in step 3.3 clu The images are combined to perform the choice operation, resulting in the image with the highest attention, represented by the feature v′. Then... With v′, and T in steps 4.1 and 4.2 enc E enc Perform MHAoA interactions separately, and include the clu from step 3.3. cen The average vector and E enc MHAoA interaction is performed to obtain the corresponding vector representation and attention weights, and finally regularization is performed to obtain the final vector.

[0148]

[0149]

[0150]

[0151]

[0152]

[0153]

[0154]

[0155]

[0156]

[0157] Among them, MHAoA masked This indicates an MHAoA layer with a mask. x represents the vector representation generated after the corresponding MHAoA operation. t ' represents the result of combining and regularizing the generated vectors at the current time. Indicates x t The result of regularization, A x Indicated by As a query, with V enc As the key value, the weight matrix obtained by MHAtt, v′ represents the weights based on A. x and P clu Image features obtained by performing the choice operation, att T ,att E They represent Two different attention weight matrices are generated after the background text and named entities interact with each other through the MHAoA module. clu represents the cluster center cen The average vector of . clu This represents the attention weights related to entity words obtained after the interaction, which will be used in the pointer network to influence entity word generation. To obtain richer weight information, visual features v′ are used as the query and T. enc Perform multi-head attention interaction to obtain the weight distribution. This distribution will be used in the subsequent word generation process.

[0158]

[0159] Output The probability distribution is transformed using softmax and used for word generation.

[0160]

[0161] Step 4.4, take the result obtained in step 4.3 att T ,att E Averaging The weight distributions a of background text words and named entity words during the generation process were obtained respectively. t and a e , where a t It contains information about fine-grained text-image interactions. Then, these two elements are mapped onto the generated vocabulary to obtain the weight distribution p on the vocabulary. T and P E , respectively, represent the generation probability distributions of background text words and named entity words.

[0162]

[0163]

[0164] P T =f(a T (35)

[0165] P E =f(a E (36) where f() represents the mapping process and softmax() represents the normalization exponential function. Decoding vectors for the dec-FITBGA module The probability values ​​p and q are calculated, representing the probability of generating a word from the background text and the probability of generating a word from the named entity list, respectively. The calculation process is as follows.

[0166]

[0167]

[0168] Among them, W p and W q These are learnable parameters, and σ() is the sigmoid function. The final probability distribution of the predicted text is... The calculation process is as follows.

[0169]

[0170] Step 5: The predicted results output by the model are compared with the true labels of the dataset using the cross-entropy loss function to calculate the loss. The NISC model is trained with the goal of minimizing the loss value to obtain the best model, which is used to generate text descriptions for news image sets.

[0171] Based on the probability distribution values ​​obtained in step 4, the NISC model is trained using the cross-entropy loss function:

[0172]

[0173] Among them, P * This represents the probability distribution of the true labels, where n represents the last time step, i.e., the length of the label text.

[0174] In practical implementation, the method proposed in this invention can be automated by those skilled in the art using computer software technology. System devices for implementing the method, such as computer-readable storage media storing the corresponding computer program of this invention and computer equipment including the computer program running the corresponding computer program, should also be within the scope of protection of this invention. For example, a corresponding system can be provided to implement news photo collection and filtering, and model training in modules.

[0175] In some possible embodiments, a news image description generation system based on graph clustering and multi-image-text interaction is provided, including the following modules:

[0176] The first module is used for data collection and data annotation. It collects news photo data from news photo websites, removes invalid information from the text, and stores the data into the dataset after annotation and proofreading.

[0177] The second module is used for data preprocessing, which normalizes the images in the generated dataset and performs named entity recognition on the text in the generated dataset to generate a list of corresponding entity words.

[0178] The third module is used for image and text feature extraction and exploration of implicit relationships between images in the atlas. The text in the preprocessed dataset is input into BERT to obtain text feature encoding, and the bag-of-words model is used to encode entity words. The images in the preprocessed dataset are input into the deep residual model to obtain image feature encoding. The graph clustering module explores the implicit relationships between images in the atlas. Principal component analysis is used to reduce the dimensionality of the extracted coarse-grained features of the images and cluster analysis is performed to mine the structured information between the images in the atlas and obtain image classification clusters and multiple cluster centers.

[0179] The fourth module is used for image-text information interaction. It combines the image clustering module with the multi-image-text interaction ITBGA module. It takes text feature encoding, image feature encoding, and structured information between images in the image set as input, supplemented by image classification cluster information and cluster centers, to complete the multi-image-text information interaction process. The decoder is used to decode the interaction results, and the weight distribution generated during the interaction process is input into the pointer network to generate the final descriptive text.

[0180] The fifth module is used for model training. It uses an end-to-end approach to train the model, employs the cross-entropy loss function, and utilizes the trained model to support the generation of descriptions for news image sets containing external text information.

[0181] In some possible embodiments, a news image description generation system based on graph clustering and multi-image-text interaction is provided, including a processor and a memory. The memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the news image description generation method based on graph clustering and multi-image-text interaction as described above.

[0182] In some possible embodiments, a news image set description generation system based on graph clustering and multi-image-text interaction is provided, including a readable storage medium on which a computer program is stored. When the computer program is executed, it implements the news image set description generation method based on graph clustering and multi-image-text interaction as described above.

[0183] In the specific implementation of news photo collection and filtering, a semi-manual annotation method can also be adopted. First, the news photo data is collected, structured, and filtered by machine. Then, manual annotation is used to supplement the data with information, and after cleaning the data, a news photo dataset is obtained.

[0184] The model is trained using a news image dataset; this example refers to the appendix. Figure 6 First, the text portion of the dataset undergoes named entity recognition using the LAC tool, and then the text is processed by a BERT pre-trained model to obtain text feature encoding. For images, after preprocessing, the data is input into a residual network to obtain image feature representations. Principal component analysis (PCA) is used to reduce the dimensionality of the coarse-grained image features, and cluster analysis is performed to extract structured information between images in the atlas, resulting in image classification clusters and multiple cluster centers. The text feature encoding, image feature encoding, and structured information between images in the atlas are then input into the ITBGA module, which combines clustering, along with image classification cluster information and cluster centers, to complete the multi-image-text information interaction process. The interaction results are decoded using a decoder, and the weight distribution generated during the interaction process is input into a pointer network to generate guide words, resulting in the final descriptive text.

[0185] The above description is merely one embodiment of the present invention and is not intended to limit the invention. Those skilled in the art can make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A method for generating news image descriptions based on graph clustering and multi-image-text interaction, comprising the following steps: Step 1, Data Collection and Labeling: Collect news photo data from news photo websites, remove invalid information from the text, label and proofread, and then store it in the dataset; Step 2, data preprocessing, normalizes the images in the generated dataset; Perform named entity recognition processing on the text in the generated dataset and generate a list of corresponding entity words; Step 3: Extract text and image features and explore the implicit relationships between images in the image set. Input the text in the preprocessed dataset into BERT to obtain text feature encoding. For entity words, use the bag-of-words model for encoding. The images in the preprocessed dataset are input into the deep residual model to obtain image feature encoding; the graph clustering module explores the implicit relationships between the images in the atlas, performs principal component analysis for dimensionality reduction and cluster analysis on the extracted coarse-grained features of the images, and mines the structured information between the images in the atlas to obtain image classification clusters and multiple cluster centers; Step 4, text-image information interaction: The graph clustering module is combined with the multi-text-image interaction ITBGA module. Text feature encoding, image feature encoding, and structured information between images in the image set are used as input, supplemented by image classification cluster information and cluster centers, to complete the multi-text-image information interaction process. The decoder is used to decode the interaction results, and the weight distribution generated during the interaction process is input into the pointer network to generate the final descriptive text. Step 5, Model Training: The model is trained in an end-to-end manner, using the cross-entropy loss function. The trained model is then used to support the generation of descriptions for news image sets containing external text information. Before the image-text interaction, the graph clustering module performs additional PCA dimensionality reduction and clustering on the image features to obtain a new image feature representation. ; Obtained through clustering The specific process is as follows: Wherein, SVD represents performing singular value decomposition to obtain the eigenvector matrix. and the square root of the eigenvalues Formula (3) represents the expression for the second dimension. The data is truncated, retaining only the first 512 dimensions; This represents the image features after PCA dimensionality reduction; kmeans indicates K-Means clustering, where the number of clusters is pre-specified, resulting in a list of cluster category IDs for each image. and the cluster center vector representation of each cluster. ; count represents the list of images in each cluster, calculated based on the list of cluster categories corresponding to each image; This means transforming the distribution of the number of images within each cluster into the corresponding weight distribution. The specific process is as follows: in, A list of weights corresponding to each cluster; The corresponding weight distribution can be directly derived from the distribution of the number of images, that is, the larger the number of images in a cluster, the greater the weight ratio.

2. The news image description generation method based on graph clustering and multi-image-text interaction as described in claim 1, characterized in that: For the news text portion, named entities are extracted, and the overall text features are obtained using the pre-trained BERT model. , represented as ,in It is the length of the text. Indicates the first The text features of each word, with the feature vector having a dimension of . ; For named entities, the Word2Vec bag-of-words model is used to obtain the first... The encoding vector of each word and position encoding vector Both dimensions are Then, the final named entity feature representation is obtained through formula (1). ,in Indicates the number of named entities:

3. The news image set description generation method based on graph clustering and multi-image-text interaction as described in claim 1, characterized in that: For those that include Image atlas Extracting ResNet152 respectively The coarse-grained and fine-grained visual features are represented as follows: ,in The first in the atlas The coarse-grained visual features of each image, with feature vectors having a dimension of . ; , ,in The first in the atlas Fine-grained visual features of an image It is the number of image regions within an image. Indicates the first in the image Fine-grained visual features of each region, with feature vector dimensions of... .

4. The news image description generation method based on graph clustering and multi-image-text interaction as described in claim 1, characterized in that: Step 4 includes the following sub-steps: 1) Input the image and text features into the first part of the clustering combined ITBGA module, namely the clustering combined CITBGA module. First, perform simple processing and transformation on the feature vectors, and then use two MHAoA structures, with image and text features as query vectors respectively. to act as each other , Information is processed through cross-attention interactions to obtain the final feature representation. Here, selfAtt represents the self-attention layer. This represents a vector concatenation operation; 2) Input the image and text features output in step 1), and the clustered image weight distribution, into the second part of the clustering combined ITBGA module, namely the enc-FITBGA module for clustering combined. Where, Avgpool represents the... Average pooling is performed to obtain the overall feature representation of the text; Therefore As a query, with As the key, the weight matrix is ​​obtained after multi-head attention (MHAtt); choice represents the image selection operation, determining the image with the highest attention, and its feature vector is represented as... LayerNorm represents a normalization operation on the input; FFN represents a two-layer feedforward network with ReLU as the activation function after the fully connected operation; Subsequent analysis of named entity features Repeated CITBGA and enc-FITBGA processes with The interaction is performed, and the final output of the overall visual features is represented as follows: Named entity features are represented as ; 3) Input the image and text features output in step 2), the clustered image weight distribution, and the graph cluster centers into the third part of the clustering combined ITBGA module, namely the clustering combined dec-FITBGA module. in, , They represent Two different attention weight matrices are generated after the background text and named entities interact with each other through the MHAoA module; Representing cluster centers The average vector; This represents the attention weights related to entity words obtained after the interaction, which will be used in the pointer network to influence entity word generation; visual features are used to obtain richer weight information. As a query and Perform multi-head attention interaction to obtain the weight distribution. ; Output The probability distribution is transformed using softmax and then used for word generation. 4) The result obtained in step 3) , , Averaging , , The weight distributions of background text words and named entity words during the generation process were obtained respectively. and ,in It includes information on fine-grained text-image interactions; then, these two are mapped onto the generated vocabulary to obtain the weight distribution on the vocabulary. and These are the generation probability distributions of background text words and named entity words, respectively. in, Represents the mapping process; the decoding vector for the dec-FITBGA module. , , , The probability value is obtained by calculation. and , representing the probability of generating a word from the background text and the probability of generating a word from the named entity list, respectively; the calculation process is as follows. in, and These are learnable parameters. It is the sigmoid function; the final probability distribution of the predicted text is... The calculation process is as follows:

5. The news image description generation method based on graph clustering and multi-image-text interaction as described in claim 1, 2, 3, or 4, characterized in that: Based on the probability distribution values ​​obtained in step 4, the NISC model is trained using the cross-entropy loss function: in, This represents the probability distribution of the true labels.

6. A news image description generation system based on graph clustering and multi-image-text interaction, characterized in that: Used to implement the news image set description generation method based on graph clustering and multi-image-text interaction as described in any one of claims 1-5; Includes the following modules, The first module is used for data collection and data annotation. It collects news photo data from news photo websites, removes invalid information from the text, and stores the data into the dataset after annotation and proofreading. The second module is used for data preprocessing, which normalizes the images in the generated dataset. Perform named entity recognition processing on the text in the generated dataset and generate a list of corresponding entity words; The third module is used for image and text feature extraction and exploration of implicit relationships between images in the image set. The text in the preprocessed dataset is input into BERT to obtain text feature encoding, and the bag-of-words model is used to encode entity words. The images in the preprocessed dataset are input into the deep residual model to obtain image feature encoding; the graph clustering module explores the implicit relationships between the images in the atlas, performs principal component analysis for dimensionality reduction and cluster analysis on the extracted coarse-grained features of the images, and mines the structured information between the images in the atlas to obtain image classification clusters and multiple cluster centers; The fourth module is used for image-text information interaction. It combines the image clustering module with the multi-image-text interaction ITBGA module. It takes text feature encoding, image feature encoding, and structured information between images in the image set as input, supplemented by image classification cluster information and cluster centers, to complete the multi-image-text information interaction process. The decoder is used to decode the interaction results, and the weight distribution generated during the interaction process is input into the pointer network to generate the guide words, so as to obtain the final descriptive text. The fifth module is used for model training. It uses an end-to-end approach to train the model, employs the cross-entropy loss function, and utilizes the trained model to support the generation of descriptions for news image sets containing external text information.

7. A device, characterized in that: It includes a processor and a memory, the memory being used to store program instructions, and the processor being used to call the stored instructions in the memory to execute the news atlas description generation method based on graph clustering and multi-graph-text interaction as described in any one of claims 1-5.

8. A readable storage medium, characterized in that: The readable storage medium stores a computer program, which, when executed, implements a news image description generation method based on graph clustering and multi-image-text interaction as described in any one of claims 1-5.

Images