A method for interpreting image paragraphs based on topic clustering and transition prediction

Through the method based on topic clustering and transfer prediction, the problem of insufficient topic accuracy and coherence in the image paragraph interpretation system is solved, the interpretation performance and interpretability are improved, and better paragraph generation effect is achieved.

CN116737976BActive Publication Date: 2024-12-06UNIV OF SCI & TECH BEIJING
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310704495.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-06-14
Publication Date
2024-12-06
Estimated Expiration
2043-06-14

AI Technical Summary

Technical Problem

The existing image paragraph interpretation system has defects in the processing of the topic accuracy of each sentence in the generated paragraph and the topic consistency between sentences, resulting in insufficient interpretation performance.

Method used

Using a method based on topic clustering and transfer prediction, the sentence topic is defined as a cluster in the sentence embedded in the space, and the clustering algorithm is used to obtain discrete topic tags, and the topic transfer probability matrix is ​​constructed, and the matrix is ​​used to interpret the hierarchical image paragraphs. The intent model is supervised and trained to enhance the semantic coherence of the model.

Benefits of technology

The interpretation performance of the image paragraph interpretation system has been significantly improved. The system's CIDEr index has been increased from 38.76 to 40.23, achieving the current optimal interpretation performance of image paragraph interpretation and improving the interpretability of system topic prediction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116737976B_ABST
    Figure CN116737976B_ABST
Patent Text Reader

Abstract

The present invention discloses an image paragraph interpretation method based on topic clustering and transfer prediction, including: defining sentence topics as clusters in sentence embedding space, clustering sentences in training data using a preset clustering algorithm, and obtaining discrete topic labels for each sentence according to the clustering results; statistically summarizing the topic labels of all training data to construct a topic transfer probability matrix; supervised training of a preset hierarchical image paragraph interpretation model based on topic labels; wherein the image paragraph interpretation model generates a semantically coherent paragraph description by utilizing the topic cohesion information contained in the topic transfer probability matrix; and processing the image to be interpreted using the trained image paragraph interpretation model to obtain the image paragraph interpretation. The technical solution of the present invention improves the interpretation performance of the image paragraph interpretation model and improves the interpretability of topic prediction.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image paragraph interpretation, and in particular to an image paragraph interpretation method based on topic clustering and transition prediction. Background Art

[0002] Image interpretation is a deep-level image understanding task proposed in recent years, which allows computers to give smooth and fluent natural sentences describing the content of the image based on the input image. It integrates computer vision, natural language processing and machine learning. Therefore, structures such as convolutional neural networks in image processing and encoder-decoder models in natural language processing are widely used in image interpretation. This task has high requirements for software systems. The system not only needs to accurately extract semantic information from the image and understand the image content to a large extent, but also needs to generate human-readable text descriptions. Early image interpretation work based on non-deep learning methods can often only generate grammatically correct but single-structured sentences. With the rapid development of deep learning, image interpretation has become a hot topic in the fields of computer vision and natural language processing. At present, image interpretation work can generate semantically accurate and detailed sentences based on images.

[0003] Most of the existing research work on image interpretation tasks is to let the system generate a single sentence to describe the image. As a branch or an extension of the image interpretation task, image paragraph interpretation aims to let the system automatically generate a semantically rich and coherent paragraph to describe the input image from different aspects. Since paragraphs have more complex structures and larger semantic capacity than sentences, image paragraph interpretation is more challenging for the system than conventional image interpretation tasks.

[0004] Existing image paragraph interpretation systems can be divided into two types: one is a hierarchical system and the other is a non-hierarchical system. The hierarchical system first extracts a high-level representation of the paragraph based on the image, and passes this high-level representation to the low-level decoder to gradually expand a paragraph to describe the image. Most hierarchical systems contain two layers of decoders. The first layer is used to extract the topic representation of each sentence in the paragraph to be generated from the image, and pass the topic representation to the next layer of decoders to generate each sentence word by word. The non-hierarchical system has a simpler structure and mostly contains only a single layer of decoders. Similar to conventional image interpretation systems, non-hierarchical systems do not generate intermediate representations of paragraphs, but directly generate an entire paragraph word by word. The two systems reflect different modeling methods for paragraphs. The hierarchical system models the paragraph as a sequence of sentences, and each sentence is a sequence of words; while the non-hierarchical system only models the paragraph as a sequence of words. The modeling method of paragraphs in hierarchical systems is closer to humans than that of non-hierarchical systems, and can often achieve better interpretation performance. Therefore, there is currently more research work on hierarchical systems than on non-hierarchical systems.

[0005] However, in the current research work on hierarchical systems, the high-level decoders of the system have the defects of difficulty in supervision and lack of cohesion information in the extraction of topic representation. Specifically, humans model topics in a discrete form, usually representing them in the form of semantically related words. But in most hierarchical systems, topics are represented by feature vectors of continuous representations generated by high-level decoders. The connection between these topic vectors and image semantics is implicit, and the continuity of topic vectors makes them difficult to be directly supervised during model training, which easily causes the model to have deviations in the process of understanding image semantics. In addition, given the continuous modeling of topics by existing hierarchical systems, these systems lack the extraction and utilization of clear topic cohesion information, which may lead to a lack of sentence coherence in the generated paragraphs. The above problems work together to affect the interpretation performance of the system.

[0006] In summary, the existing image paragraph interpretation system has shortcomings in the topic accuracy of each sentence in the generated paragraph and the topic coherence between sentences, and there is a lot of room for improvement in the interpretation performance. Summary of the invention

[0007] The present invention provides an image paragraph interpretation method based on topic clustering and transition prediction to solve the technical problems that the existing image paragraph interpretation system has vacancies in processing the topic accuracy of each sentence in the generated paragraph and the topic coherence between sentences.

[0008] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0009] On the one hand, the present invention provides an image paragraph interpretation method based on topic clustering and transition prediction, and the image paragraph interpretation method based on topic clustering and transition prediction comprises:

[0010] Define sentence topics as clusters in the sentence embedding space, use a preset clustering algorithm to cluster the sentences in the training data, and obtain discrete topic labels for each sentence based on the clustering results;

[0011] Perform statistical induction on the topic tags of all training data and construct a topic transition probability matrix;

[0012] Performing supervised training on a preset hierarchical image paragraph interpretation model based on topic tags; wherein the image paragraph interpretation model generates a semantically coherent paragraph description by utilizing topic cohesion information contained in the topic transition probability matrix;

[0013] The trained image paragraph interpretation model is used to process the image to be interpreted to obtain the image paragraph interpretation.

[0014] Furthermore, the sentence topic is defined as a cluster in the sentence embedding space, and the sentences in the training data are clustered using a clustering algorithm. According to the clustering results, the discrete topic label of each sentence is obtained, including:

[0015] Use the pre-trained language model to extract sentence embeddings for all sentences in the training data;

[0016] Use the preset dimensionality reduction algorithm to reduce the dimension of all sentence embeddings;

[0017] Use the preset clustering algorithm to cluster the sentence embeddings after dimensionality reduction. During clustering, the silhouette coefficient is used to measure the consistency of the clustering results and search for the optimal value of the number of clusters.

[0018] According to the clustering results, sentences on the same topic are connected into a long document, and the long document is input into a preset keyword extraction model to extract keywords;

[0019] Based on the keyword extraction results, visualize them with the highest ranking keywords for each topic.

[0020] Furthermore, the topic tags of all training data are statistically summarized to construct a topic transition probability matrix, including:

[0021] Calculate the number of each adjacent topic pair in all training data paragraphs to build a topic transition frequency matrix M fre ∈Z n×n , where n represents the total number of topics; M fre The element in row i and column j of is the total number of transfers from topic i to topic j;

[0022] M fre The elements of and the number of occurrences of the corresponding topics are logarithmically normalized to obtain the normalized matrix M norm ∈R n ×n , the calculation formula is as follows:

[0023]

[0024] in, Indicates M norm The element in the i-th row and j-th column of N j is the number of occurrences of topic j in the training data;

[0025] Further normalize M norm Get the topic transfer probability matrix M∈R n×n , the calculation formula is as follows:

[0026]

[0027] Among them, M ij Represents the element in the i-th row and j-th column of M, where the i-th row of M represents the conditional probability distribution of the topic immediately following the current topic i; M is represented by a directed graph, wherein, in the directed graph, topics are nodes and topic transition probabilities are edge weights.

[0028] Furthermore, the image paragraph interpretation model includes: a Faster R-CNN module based on ResNet-101, an encoder including an X-Linear attention module, a sentence-level decoder, and a word-level decoder; wherein,

[0029] The Faster R-CNN module is used to extract visual object features from the input image;

[0030] The encoder is used to obtain enhanced region-level and image-level features based on the visual object features extracted by the Faster R-CNN module, and input the enhanced region-level and image-level features into the sentence-level decoder and the word-level decoder;

[0031] The topic features output by the sentence-level decoder guide the word-level decoder to generate sentences that are semantically consistent with the predicted topic; the language context information of the word-level decoder at the moment each sentence is generated is returned to the sentence-level decoder to help it generate the topic of the next sentence; the sentence-level decoder also uses the topic transition probability matrix as a human topic transition prior to enhance the coherence of its predicted topic.

[0032] Further, the sentence-level decoder includes a graph convolutional neural network, which updates the embedding representation of the topic by propagating the transition probability between topics based on the topic transition probability matrix;

[0033] The graph convolutional neural network embeds all topics into a representation matrix And the topic transition probability matrix M∈R n×n As input, and update the embedding representation of the topic to Among them, d (l-1) is the dimension of the embedding representation of the input topic, d (l) is the dimension of the embedded representation of the updated topic;

[0034] The update formula for the topic embedding representation is as follows:

[0035] E (l) =f(M(E (l-1) )W)

[0036] in, is the parameter of the fully connected layer, and f(.) is the LeakyRelu activation function.

[0037] Furthermore, when the word-level decoder completes the generation of a sentence, the cell state s of the long short-term memory network LSTM of the word-level decoder word is input into the LSTM of the sentence-level decoder as language context information, which contains the long-term memory of previously generated sentences; global image-level features Topic characteristics of the last moment And the embedding vector E of the topic at the previous moment (0) [topic t-1 ] will be input into LSTM as shown below:

[0038]

[0039] in, Representation of LSTM TSP The output of LSTM TSP represents the long short-term memory network in the sentence-level decoder, represents the output of the long short-term memory network in the sentence-level decoder at the previous moment, s word Represents the cell state of the long short-term memory network of the word-level decoder.

[0040] Furthermore, the sentence-level decoder is equipped with an X-Linear attention module;

[0041] The working process of the sentence level decoder includes:

[0042] Embed the topic of the last moment after the graph convolutional neural network is updated and LSTM TSP Output Make a connection and send the connection result to the fully connected layer to get the original logits, as shown in the following formula:

[0043]

[0044] Among them, A and b are both trainable parameters;

[0045] Logits raw It is input as a query into the X-Linear attention module, where the keys and values ​​of the X-Linear attention module are the enhanced region-level features obtained from the encoder.

[0046] The attention features and logits output by the X-Linear attention module raw The connection is then input into the linear layer and the gated linear unit GLU to obtain the topic features of the current moment. The obtained topic features are used to predict topics through a softmax layer.

[0047] Furthermore, the training of the image paragraph interpretation model includes a supervised learning stage and a reinforcement learning stage; a hierarchical supervised learning method is used in both stages; in the supervised learning stage, the sentence-level decoder is supervised by topic tags obtained by topic clustering, and the word-level decoder is supervised by the KL divergence loss between word probability output and ground truth description; in the reinforcement learning stage, both the sentence-level decoder and the word-level decoder are trained using the self-critical sequence training (SCST) training strategy.

[0048] Furthermore, the overall loss function L1 of the supervised learning stage is as follows:

[0049] L1=L kl +λL RR ;

[0050]

[0051]

[0052] Among them, λ is a hyperparameter used to adjust the supervision strength of the sentence-level decoder;

[0053] N is the number of paragraphs, Represents the number of words in the i-th paragraph;

[0054]

[0055] in, is the predicted probability distribution of the jth word in the ith paragraph, is the smoothed distribution of the jth word label in the ith paragraph, and v is the number of words in the vocabulary;

[0056] Indicates the number of sentences in the i-th paragraph;

[0057]

[0058] in, is the topic prediction probability distribution of the kth sentence in the ith paragraph, is the probability corresponding to topic y; SET ik is the set of subsequent topics of the kth sentence in the ith paragraph.

[0059] Furthermore, in the reinforcement learning stage, the SCST strategy is used to optimize the multi-metric scores of the word sequence of the word-level decoder; the multi-metric score is obtained by linearly combining CIDEr, METEOR and BLEU-4 in a ratio of 1:0.5:0.5; its training goal is to minimize the negative expected score corresponding to the model parameter θ; the sentence-level decoder is optimized using the Rouge score of the topic sequence, and its training goal is to minimize the opposite of the Rouge expectation corresponding to the model parameter θ;

[0060] The total loss function of the reinforcement learning stage is as follows:

[0061]

[0062] in, represents the SCST loss corresponding to the word-level decoder, represents the SCST loss corresponding to the sentence-level decoder, and β is a hyperparameter used to adjust the supervision strength of the sentence-level decoder.

[0063] On the other hand, the present invention further provides an electronic device, comprising a processor and a memory; wherein the memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the above method.

[0064] In yet another aspect, the present invention further provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, and the instruction is loaded and executed by a processor to implement the above method.

[0065] The beneficial effects brought about by the technical solution provided by the present invention include at least:

[0066] 1. Improved the interpretation performance of the image paragraph interpretation system; the technical solution of the present invention improves the interpretation performance of the image paragraph interpretation system from multiple angles. First, the topic clustering method is used to obtain discrete topic tags, and the topic tags are used to supervise the training of the sentence-level TSP decoder to make the topic prediction accurate. Secondly, the training data is used to obtain the human prior of topic transfer, and the prior is used to explicitly let the model predict more coherent and reasonable topics. Combining the above optimizations, the system CIDEr index is improved from 38.76 to 40.23, reaching the current optimal image paragraph interpretation performance.

[0067] 2. Improved the interpretability of the system's topic prediction: To make topics interpretable, the technical solution of the present invention concatenates sentences of the same topic into a long document and inputs it into the KeyBERT model to extract keywords. Therefore, each topic can be visualized using its highest-ranked keywords. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0069] Figure 1 It is the overall framework and main training process of the image paragraph interpretation system provided by the embodiment of the present invention;

[0070] Figure 2 is a schematic diagram of a topic clustering method provided by an embodiment of the present invention;

[0071] Figure 3 is a topic transfer schematic diagram provided by an embodiment of the present invention;

[0072] Figure 4 It is a detailed structural diagram of a double-layer decoder provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0073] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0074] First embodiment

[0075] This embodiment provides an image paragraph interpretation method based on topic clustering and transition prediction, including processing training data, improving training methods, and building a new model, aiming to solve the following technical problems:

[0076] (1) The continuous modeling of topics is contrary to the discrete topics that humans know and has poor interpretability

[0077] For humans, topics are usually represented in the form of semantically related words. However, in most hierarchical systems, topics are represented by continuous feature vectors generated by sentence-level decoders. Such topic representation is significantly different from the discrete topic representation in human cognition and has poor interpretability. The topics extracted by the system constructed in this embodiment are in a discrete form, which can be well explained using words, thereby improving the interpretability of the system.

[0078] (2) Difficulty in topic monitoring

[0079] The connection between continuous topic vectors and image semantics is implicit, which increases the difficulty of obtaining topic labels, making it impossible for most hierarchical systems to perform direct supervised training in the topic prediction stage. This makes it more difficult to train a hierarchical system with good interpretation performance. However, this embodiment uses a topic clustering method to cluster sentences in the training data to obtain discrete topic labels for each sentence, and uses the topic labels to supervise the hierarchical system of this embodiment to obtain a hierarchical image paragraph interpretation system with excellent performance.

[0080] (3) The system lacks the ability to explicitly extract and utilize topic cohesion information

[0081] When humans organize a description, they will consider the order of all sentences in the paragraph at the topic level and try to make the sentences as coherent as possible. Most current methods lack consideration of sentence coherence. We constructed a topic transition probability matrix by statistically summarizing the topic tags of all training data. The system generates semantically coherent paragraph descriptions by using the topic cohesion information contained in the matrix.

[0082] For problem (1), in our technical solution, topic clustering will be used as the basis. Through topic clustering, we can obtain topic tags to complete the discrete modeling of topics, and use the KeyBERT model to interpret each discrete topic. For problem (2), the present invention uses these discrete topic tags to directly supervise the model training process, which enables the word-level decoder to generate sentences with clear semantic focus. For problem (3), the topic transfer probability matrix obtained according to the topic clustering results can reflect the pattern of human topic transfer in a statistical sense. The system will combine the matrix, the current paragraph generation status and visual information to make reasonable predictions on subsequent topics, thereby effectively enhancing the logical consistency of the generated paragraphs. Through the above improvements, the method of this embodiment can effectively improve the quality of generated paragraphs.

[0083] Based on the above, the method of this embodiment includes the following steps:

[0084] S1, define sentence topics as clusters in the sentence embedding space, use a preset clustering algorithm to cluster the sentences in the training data, and obtain discrete topic labels for each sentence based on the clustering results;

[0085] S2, statistically summarize the topic tags of all training data and construct a topic transition probability matrix;

[0086] S3, performing supervised training on a preset hierarchical image paragraph interpretation model based on topic tags; wherein the image paragraph interpretation model generates a semantically coherent paragraph description by utilizing the topic cohesion information contained in the topic transition probability matrix;

[0087] S4, using the trained image paragraph interpretation model to process the image to be interpreted to obtain the image paragraph interpretation.

[0088] Specifically, Figure 1 The overall framework of our system and the main training process are shown. Faster R-CNN based on ResNet-101 is first used to extract visual object features from the input image. These features are input into the encoder containing the X-Linear attention module to obtain enhanced regional and image-level features. These features are then fed into the sentence-level decoder and the word-level decoder. The two levels of decoders work in a collaborative way. On the one hand, the topic features output by the sentence-level decoder guide the word-level decoder to generate sentences that are semantically consistent with the predicted topic. On the other hand, the language context information of the word-level decoder at the moment when each sentence is generated is returned to the sentence-level decoder to help it generate the topic of the next sentence. In addition, the sentence-level decoder will additionally use the topic transition probability matrix as a human topic transition prior to enhance the coherence of its predicted topic.

[0089] Similar to other works on image paragraph paraphrase, our proposed system is trained using two stages, including a supervised learning stage and a reinforcement learning stage. In addition, we adopt a hierarchical supervised learning approach. In the supervised learning stage, we design a loss function called Range Reduction, which encourages the sentence-level decoder to generate any topic of the subsequent sentence. At the same time, we supervise the word-level decoder using the KL divergence loss calculated between the word probability output and the ground truth. In the reinforcement learning stage, both decoders are trained using the SCST training strategy (self-critical sequence training strategy).

[0090] Below, we first describe the topic clustering method that is the basis of our technical solution; then we introduce the details of topic transition prediction; then we introduce our two-stage hierarchical training method; and finally, we give an overall discussion of the reasoning process of our system.

[0091] 1. Topic Clustering

[0092] In most existing hierarchical image paragraph interpretation systems, the feature vectors generated by the sentence-level decoder are often referred to as sentence topics. However, given that the connection between these continuous topic vectors and image semantics is implicit, this topic representation lacks interpretability. In addition, under such a definition, it is difficult to assign a training-usable label to a sentence, which makes it difficult to directly supervise these topic vectors. Considering human cognitive habits, we believe that a better representation of sentence topics should be discrete, clear, and interpretable. This not only makes it easier to design supervisory signals for the sentence-level decoder, but also greatly increases the interpretability of the system's paragraph generation process. Based on this understanding, we propose to define sentence topics as clusters in the sentence embedding space. Our topic acquisition process is as follows Figure 2 shown.

[0093] Specifically, we first use the pre-trained language model RoBERTa to extract sentence embeddings for all sentences in the training set. Considering that high dimensions may bring difficulties to most clustering algorithms, we reduce all sentence embeddings from 768 dimensions to 10 dimensions using UMAP (Uniform Manifold Approximation and Projection for Dimension Reduction). UMAP is an effective dimensionality reduction method that maintains the manifold structure of high-dimensional space in low-dimensional space. Finally, we use the K-means clustering algorithm to cluster the sentence embedding representations after dimensionality reduction. Intuitively, sentences belonging to the same cluster are semantically closer than sentences from different clusters. Therefore, in our method, sentences in the same cluster are considered to have the same topic. We use the silhouette score to measure the consistency of the clustering results and search for the optimal value of the number of clusters. One of the samples x i The calculation of the silhouette coefficient is shown in formula (1), where A(x i ) is the average intra-cluster distance, B(x i ) is x i The minimum average dissimilarity of a vector to other clusters.

[0094]

[0095] By calculating the average silhouette coefficient of all samples, we can get the consistency evaluation of the overall clustering results, as shown in formula (2). Where m is the total number of samples to be clustered. The value, the number of clusters selected for the Stanford image paragraph dataset is 81.

[0096]

[0097] To make the topic interpretable, we concatenate sentences of the same topic into a long document and feed it into the KeyBERT model to extract keywords. This way we can visualize each topic with its top-ranked keyword, such as Figure 1 and Figure 3 As shown. Among them, Figure 3 The left half is a topic transition directed graph, and the right half is an image and its corresponding paragraph ground truth. The topics, keywords, and transition probabilities in the figure are all extracted by the present invention.

[0098] After topic clustering, each sentence in the training set is assigned a topic label that can be used for supervision during training. Such topic labels are discrete in representation, have clear semantics, and can be interpreted using keywords. In addition, because these topic labels are obtained by using off-the-shelf open source language models and unsupervised clustering methods, no manual annotation is required. In addition, unlike other topic extraction methods such as LDA (Latent Dirichlet Allocation) and NMF (Nonnegative Matrix Factorization), our method is not limited by the length of the text and does not require excessive tuning of hyperparameters. Experiments have shown that using these topic labels during the training process of our system can achieve the current state-of-the-art image paragraph interpretation performance.

[0099] 2. Topic shift prediction

[0100] A paragraph is not a collection of sentences. The order of sentences in a paragraph reflects the description logic and expression habits. However, some previous methods explicitly ignore the order of sentences when generating paragraphs. In our system, we explicitly consider the modeling and prediction of sentence order to facilitate the generation of more logical and semantically consistent paragraphs. Specifically, we propose Figure 4 The sentence-level TSP (Topic Shift Prediction) decoder shown in FIG. 1 solves the sentence-level sorting problem.

[0101] Visual information, language context information, and human topic transition priors are fully utilized by the sentence-level TSP decoder to predict more reasonable topic sequences.

[0102] First, let’s introduce how to construct the topic transition probability matrix. After the topic clustering is completed, each sentence in the training set is assigned a topic label. We first calculate the number of each adjacent topic pair in the paragraph to build a topic transition frequency matrix M. fre ∈Z n×n , where n is the total number of topics. The matrix M fre The element in row i and column j of is the total number of transfers from topic i to topic j. In order to reduce the impact of topic frequency, we fre The elements of and the number of occurrences of the corresponding topics are logarithmically normalized to obtain the normalized matrix M norm ∈R n×n , the calculation method is shown in Formula 3, where N j is the number of occurrences of topic j in the training set.

[0103]

[0104] Finally, as in Formula 4, M is further normalized norm Get the topic transfer probability matrix M∈R n×n .

[0105]

[0106] The i-th row of M can be regarded as the conditional probability distribution of the topic immediately following the current topic i. M can be represented by a directed graph, where topics are nodes and transition probabilities are edge weights, as shown in the following example: Figure 3 Shown on the left.

[0107] The module that uses the topic transition probability matrix in the model is GCN (Graph Convolutional Networks). Its specific location is in the sentence-level TSP decoder, which updates the embedding representation of the topic by propagating the transition probability between topics. The GCN we use has two layers (l = 1, 2). The lth layer embeds the matrix of all topics And the topic transition probability matrix M∈R n×n as input and update the embedding to where d (l-1) is the dimension of the topic embedding input to this layer, d (l) is the dimension of the updated topic embedding. The mathematical formula of the GCN layer is shown in formula (5), where is the parameter of the fully connected layer, and f(.) is the LeakyRelu activation function.

[0108] E (l) =f(M(E (l-1) )W)#(5)

[0109] like Figure 4 As shown, E (0) is the initial embedding matrix for all topics a, E (2) is the final updated topic embedding representation. (0) , E (2) It can better reflect the relationship between topics. Finally, we use the updated topic embedding of the previous moment To predict the next topic.

[0110] from Figure 3 As can be seen from M, the transition probability between different topics is relatively low. This reveals the huge diversity of human topic transfer, indicating that even with the help of human priors contained in M, obtaining a reasonable and coherent topic sequence for the input image is still a very challenging task. In a large number of existing works, the prediction of topics relies solely on the visual information extracted from the input image. The drawback of this strategy is that using fixed topics to guide the word-level decoder may sacrifice the flexibility of sentence generation. For example, if the previous sentence does not fully describe the relevant content of a topic, it is better to continue the topic when generating subsequent sentences instead of changing the topic hastily.

[0111] Therefore, we propose to input the language context information generated by the word-level decoder into the sentence-level TSP decoder to help topic prediction. The language context information reflects the real-time state of the word-level decoder. Therefore, when predicting subsequent topics, the sentence-level TSP decoder not only considers visual information and human priors, but also considers the generation of the currently generated paragraph, thereby producing more semantically consistent and reasonable paragraph generation results.

[0112] Specifically, when the word-level decoder completes the generation of a sentence, the cell state s of its LSTM (Long-Short Term Memory) word It is input into the LSTM of the sentence-level TSP decoder as a language context feature, which contains the long-term memory of the previously generated completed sentences. Global image-level features Topic characteristics of the last moment And the embedding vector E of the topic at the previous moment (0) [topic t-1] will be input into LSTM, as shown in formula (6):

[0113]

[0114] Among them, LSTM TSP represents the long short-term memory network in the sentence-level decoder, represents the output of the long short-term memory network in the sentence-level decoder at the previous moment, s word Represents the cell state of the long short-term memory network of the word-level decoder.

[0115] like Figure 4 As shown in Figure 2, the proposed TSP decoder is equipped with an X-Linear attention module. We embed the topic of the previous moment updated by GCN and LSTM TSP Output Make a connection. Send the connection result to the fully connected layer to get the original logits, as shown in formula (7). Among them, A and b are trainable parameters.

[0116]

[0117] logits raw It contains visual information, topic transfer patterns, previous topic information, and language context information. It will be input as a query into the X-Linear attention block. The keys and values ​​of this attention block are the enhanced region-level features obtained from the image encoder. We take the attention features and logits output by the attention block raw Connect and then input into the linear layer and GLU (Gated Linear Unit) to get the topic features of the current moment The topic feature is used to predict the topic through a softmax layer.

[0118] like Figure 4 As shown in the figure, the word-level decoder is similar to the sentence-level TSP decoder, except that it does not contain the GCN layer and the subsequent FC (fully connected) layer.

[0119] 3. Hierarchical supervision

[0120] The proposed image paragraph interpretation model will be trained in two stages. These two stages are the supervised learning stage and the reinforcement learning stage. In both stages, we use hierarchical supervision. The sentence-level TSP decoder is supervised by the topic labels obtained by topic clustering, and the word-level decoder is supervised by the ground truth paragraph description. In the supervised learning stage, we supervise the word-level decoder using the KL divergence loss between the word probability output and the ground truth description, as shown in formula (8). is the predicted probability distribution of the jth word in the ith paragraph, is the smoothed distribution of the jth word label in the ith paragraph, and v is the number of words in the vocabulary.

[0121]

[0122] The overall KL divergence loss is shown in formula (9), where N is the number of paragraphs, is the number of words in the i-th paragraph.

[0123]

[0124] In the supervised learning stage, we introduced a new loss function called Range Reduction (RR). This loss function involves the concept of subsequent topics. In a paragraph, the subsequent topic of a sentence refers to the topic label corresponding to the sentence and all subsequent sentences. The RR loss is defined as the inverse of the sum of the predicted probabilities of all topics in the subsequent topic set, as shown in formula (10), where is the topic prediction probability distribution of the kth sentence in the ith paragraph, is the probability corresponding to topic y. SET ik is the set of subsequent topics of the kth sentence in the ith paragraph.

[0125]

[0126] The overall RR loss is shown in formula 11, where is the number of sentences in the ith paragraph.

[0127]

[0128] In fact, RR loss is a looser way of supervising the sentence-level TSP decoder. We have experimentally verified that using strict topic supervision such as cross-entropy loss will harm the training effect of the model. The reason may be that in the early stages of training, it is very difficult to strictly optimize the topic-level TSP decoder and the word-level decoder at the same time, which leads to unsatisfactory training results. Unlike the cross-entropy loss function that guides the model to predict a specific topic, the RR loss function allows the model to generate a set of subsequent topics. At each time step, the model is encouraged to increase the probability of any one or several topics in the set of subsequent topics, rather than specifying one of the topics. This is similar to the flexibility of human subsequent topic selection.

[0129] The overall loss function in the supervised learning phase is the KL divergence loss L as shown in formula (12): kl and RR loss L RR where λ is a hyperparameter used to adjust the supervision strength of the sentence-level TSP decoder.

[0130] L1=L kl +λL RR #(12)

[0131] In the reinforcement learning stage, we use the SCST strategy to optimize the multi-metric score of the word sequence of the word-level decoder. This score is obtained by linearly combining CIDEr, METEOR, and BLEU-4 in a ratio of 1:0.5:0.5, aiming to obtain a balanced performance. The training goal is to minimize the negative expected score corresponding to the model parameter θ, as shown in formula (13), where r word (ω) is the multi-metric score of the sampled paragraph ω, E ω~θ represents the expected score corresponding to the model parameter θ.

[0132]

[0133] The gradient of can be approximated using formula (14), where r word (ω S )and is the excitation of the randomly sampled paragraph and the excitation of the maximally sampled paragraph, represents the gradient, p(w S ) represents the probability corresponding to the randomly sampled paragraph.

[0134]

[0135] The sentence-level TSP decoder is optimized using the Rouge score of the topic sequence. Its goal is to minimize the inverse of the Rouge expectation corresponding to the model parameter θ, as shown in formula (15), where r topic (t) is the Rouge score of the sampled topic sequence t, E t~θ represents the expected score corresponding to the model parameter θ. The gradient of this loss function can be approximated in a manner similar to formula (14).

[0136]

[0137] The total loss function in the reinforcement learning phase is the word SCST loss With topic SCST loss is a linear combination of , as shown in Formula 16, where β is a hyperparameter used to adjust the supervision strength of the sentence-level TSP decoder.

[0138]

[0139] 4. System Reasoning

[0140] Here we introduce the reasoning process of the whole system in general. Figure 1 As shown in the figure, for an input image, Faster R-CNN first extracts object features in the image. The encoder uses the object features as input to obtain a set of enhanced region-level features and image-level features. The sentence-level TSP decoder uses these enhanced visual features, the topic transition probability matrix, and the language context features to predict the topic of the sentence to be generated. The topic features are then fed into the word-level decoder for word prediction in the sentence. The language context features are initialized to an all-zero vector when generating the topic of the first sentence of a paragraph, because the word-level decoder has not yet participated in the generation of this paragraph. Then, the word-level decoder generates a sentence word by word using the topic features and enhanced visual features. When the last word of the sentence is generated, the word-level decoder provides a language context feature to the sentence-level TSP decoder as feedback to help it predict the topic of the next sentence. The above process is repeated until the word-level decoder outputs a paragraph end marker indicating that the paragraph generation is complete.

[0141] In summary, this embodiment provides an image paragraph interpretation method based on topic clustering and transition prediction. The image paragraph interpretation system constructed by this method explicitly considers the transfer characteristics and discrete characteristics of the topic, and the two-layer decoders cooperate with each other to jointly complete the generation of clear and coherent paragraphs. In addition, this method uses topic clustering to obtain topic tags, which solves the problem of difficult topic prediction supervision, and proposes a new RR loss function to help the model perform better training in a loose manner. Finally, a two-stage hierarchical method is used to train the model to achieve the highest interpretation performance among all models of the same period.

[0142] Second embodiment

[0143] This embodiment provides an electronic device, which includes a processor and a memory; wherein the memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the method of the first embodiment.

[0144] The electronic device may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) and one or more memories, wherein the memory stores at least one instruction, and the instruction is loaded by the processor to execute the above method.

[0145] Third embodiment

[0146] This embodiment provides a computer-readable storage medium, which stores at least one instruction, and the instruction is loaded and executed by a processor to implement the method of the first embodiment. The computer-readable storage medium may be a ROM, a random access memory, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc. The instructions stored therein may be loaded by a processor in a terminal to execute the method.

[0147] In addition, it should be noted that the present invention can be provided as a method, an apparatus or a computer program product. Therefore, the embodiments of the present invention can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the embodiments of the present invention can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0148] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0149] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device, so that a series of operation steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable terminal device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0150] It should also be noted that, in this article, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0151] Finally, it should be noted that the above is a preferred embodiment of the present invention. It should be pointed out that although the preferred embodiment of the present invention has been described, for those skilled in the art, once the basic creative concept of the present invention is known, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications should also be regarded as the protection scope of the present invention. Therefore, the attached claims are intended to be interpreted as including the preferred embodiment and all changes and modifications that fall within the scope of the embodiments of the present invention.

Claims

1. A method for interpreting image paragraphs based on topic clustering and transition prediction, characterized in that: include: Define sentence topics as clusters in the sentence embedding space, use a preset clustering algorithm to cluster the sentences in the training data, and obtain discrete topic labels for each sentence based on the clustering results; Perform statistical induction on the topic tags of all training data and construct a topic transition probability matrix; Performing supervised training on a preset hierarchical image paragraph interpretation model based on topic tags; wherein the image paragraph interpretation model generates a semantically coherent paragraph description by utilizing topic cohesion information contained in the topic transition probability matrix; Use the trained image paragraph interpretation model to process the image to be interpreted to obtain the image paragraph interpretation; Perform statistical induction on the topic tags of all training data and construct a topic transition probability matrix, including: Calculate the number of each adjacent topic pair in all training data paragraphs to build a topic transition frequency matrix M fre ∈Z n×n , where n represents the total number of topics; M fre The element in row i and column j of is the total number of transfers from topic i to topic j; M fre The elements of and the number of occurrences of the corresponding topics are logarithmically normalized to obtain the normalized matrix M norm ∈R n×n , the calculation formula is as follows: in, Indicates M norm The element in the i-th row and j-th column of N j is the number of occurrences of topic j in the training data; Further normalize M norm Get the topic transfer probability matrix M∈R n×n , the calculation formula is as follows: Among them, M ij represents the element in the i-th row and j-th column of M, where the i-th row of M represents the conditional probability distribution of the topic immediately following the current topic i; M is represented by a directed graph, where topics are nodes and topic transition probabilities are edge weights; The image paragraph interpretation model includes: a Faster R-CNN module based on ResNet-101, an encoder including an X-Linear attention module, a sentence-level decoder and a word-level decoder; wherein, The Faster R-CNN module is used to extract visual object features from the input image; The encoder is used to obtain enhanced region-level and image-level features based on the visual object features extracted by the Faster R-CNN module, and input the enhanced region-level and image-level features into the sentence-level decoder and the word-level decoder; The topic features output by the sentence-level decoder guide the word-level decoder to generate sentences that are semantically consistent with the predicted topic; the language context information of the word-level decoder at the moment each sentence is generated is returned to the sentence-level decoder to help it generate the topic of the next sentence; the sentence-level decoder also uses the topic transition probability matrix as a human topic transition prior to enhance the coherence of its predicted topic.

2. The image paragraph interpretation method based on topic clustering and transition prediction as claimed in claim 1, characterized in that: Define sentence topics as clusters in the sentence embedding space, use a preset clustering algorithm to cluster the sentences in the training data, and obtain discrete topic labels for each sentence based on the clustering results, including: Use the pre-trained language model to extract sentence embeddings for all sentences in the training data; Use the preset dimensionality reduction algorithm to reduce the dimension of all sentence embeddings; Use the preset clustering algorithm to cluster the sentence embeddings after dimensionality reduction. During clustering, the silhouette coefficient is used to measure the consistency of the clustering results and search for the optimal value of the number of clusters. According to the clustering results, sentences on the same topic are connected into a long document, and the long document is input into a preset keyword extraction model to extract keywords; Based on the keyword extraction results, visualize them with the highest ranking keywords for each topic.

3. The image paragraph interpretation method based on topic clustering and transition prediction as claimed in claim 1, characterized in that: The sentence-level decoder comprises a graph convolutional neural network, which updates the embedding representation of topics by propagating transition probabilities between topics based on the topic transition probability matrix; The graph convolutional neural network embeds all topics into a representation matrix And the topic transition probability matrix M∈R n×n As input, and update the embedding representation of the topic to Among them, d (l-1) is the dimension of the embedding representation of the input topic, d (l) is the dimension of the embedded representation of the updated topic; The update formula for the topic embedding representation is as follows: HAVE BEEN (l) =f(M(E (l-1) )W); in, is the parameter of the fully connected layer, and f(.) is the LeakyRelu activation function.

4. The image paragraph interpretation method based on topic clustering and transition prediction as claimed in claim 3, characterized in that: When the word-level decoder completes the generation of a sentence, the cell state s of the long short-term memory network LSTM of the word-level decoder word is input into the LSTM of the sentence-level decoder as language context information, which contains the long-term memory of previously generated sentences; global image-level features Topic characteristics of the last moment And the embedding vector E of the topic at the previous moment (0) [topic t-1 ] will be input into LSTM as shown below: in, Representation of LSTM TSP The output of LSTM TSP represents the long short-term memory network in the sentence-level decoder, represents the output of the long short-term memory network in the sentence-level decoder at the previous moment, s word Represents the cell state of the long short-term memory network of the word-level decoder.

5. The image paragraph interpretation method based on topic clustering and transition prediction as claimed in claim 4, characterized in that: The sentence-level decoder is equipped with an X-Linear attention module; The working process of the sentence level decoder includes: Embed the topic of the last moment after the graph convolutional neural network is updated and LSTM TSP Output Make a connection and send the connection result to the fully connected layer to get the original logits, as shown in the following formula: Among them, A and b are both trainable parameters; Logits raw It is input as a query into the X-Linear attention module, where the keys and values ​​of the X-Linear attention module are the enhanced region-level features obtained from the encoder. The attention features and logits output by the X-Linear attention module raw The connection is then input into the linear layer and the gated linear unit GLU to obtain the topic feature f at the current moment. t topic ; The obtained topic features are used to predict topics through a softmax layer.

6. The image paragraph interpretation method based on topic clustering and transition prediction as claimed in claim 1, characterized in that: The training of the image paragraph interpretation model includes a supervised learning stage and a reinforcement learning stage; hierarchical supervised learning is used in both stages; in the supervised learning stage, the sentence-level decoder uses topic labels obtained by topic clustering for supervision, and the word-level decoder uses the KL divergence loss between word probability output and ground truth description for supervision; In the reinforcement learning stage, both the sentence-level decoder and the word-level decoder are trained using the self-critical sequence training (SCST) training strategy.

7. The image paragraph interpretation method based on topic clustering and transition prediction as claimed in claim 6, characterized in that: The overall loss function L1 of the supervised learning stage is as follows: L1=L kl +λL RR ; Among them, λ is a hyperparameter used to adjust the supervision strength of the sentence-level decoder; N is the number of paragraphs, Represents the number of words in the i-th paragraph; in, is the predicted probability distribution of the jth word in the ith paragraph, is the smoothed distribution of the jth word label in the ith paragraph, and v is the number of words in the vocabulary; Indicates the number of sentences in the i-th paragraph; in, is the topic prediction probability distribution of the kth sentence in the ith paragraph, is the probability corresponding to topic y; SET ik is the set of subsequent topics of the kth sentence in the ith paragraph.

8. The image paragraph interpretation method based on topic clustering and transition prediction as claimed in claim 6, characterized in that: In the reinforcement learning stage, the SCST strategy is used to optimize the multi-metric score of the word sequence of the word-level decoder; the multi-metric score is obtained by linearly combining CIDEr, METEOR and BLEU-4 in a ratio of 1:0.5:0.5; its training goal is to minimize the negative expected score corresponding to the model parameter θ; the sentence-level decoder is optimized using the Rouge score of the topic sequence, and its training goal is to minimize the opposite number of the Rouge expectation corresponding to the model parameter θ; The total loss function of the reinforcement learning stage is as follows: in, represents the SCST loss corresponding to the word-level decoder, represents the SCST loss corresponding to the sentence-level decoder, and β is a hyperparameter used to adjust the supervision strength of the sentence-level decoder.