A weakly supervised cross-modal semantic coherence restoration method

Abstract

The present invention discloses a weakly supervised cross-modal semantic coherence restoration method. First, a semantic coherence restoration model is trained for each of the text and the image to obtain the text position matrix A. t , image position matrix B t , and then with the help of the cross-modal semantic alignment capability of the CLIP pre-trained model, the relative position information of the other modality is introduced into the target modality. In order to maximize the use of cross-modal relative position information, the present invention further designs an iterative training and reasoning strategy on this basis: this method alternately freezes the text and image models during the training process, and introduces the cross-modal relative position information in the frozen model into the unfrozen model through cross-modal semantic alignment. This strategy is also applied to the reasoning stage of the model, and finally topological sorting is used to obtain the text and image sorting. Relevant experiments conducted on the SIND and TACoS‑Ordering datasets demonstrate the effectiveness of the present invention, that is, the relative position information predicted by another modality can be used to restore cross-modal semantic coherence without the need for additional position information annotation.

Description

Technical Field

[0001] The present invention belongs to the technical field of semantic coherence restoration, and more specifically, relates to a weakly supervised cross-modal semantic coherence restoration method. Background Art

[0002] Semantic coherence restoration is one of the basic and common tasks in modeling document coherence, which aims to restore a set of disordered sentences (texts) into a coherent text by analyzing their semantics.

[0003] Existing semantic coherence restoration methods typically use an encoder-decoder framework with a pointer network to map text into feature vectors and align the order for prediction. On the one hand, existing semantic coherence restoration methods are mainly divided into autoregressive methods and non-autoregressive methods. Among them, autoregressive methods use recurrent neural networks to map text representations into feature vectors. However, this approach can capture incorrect semantic logic between sentences, thereby misleading the decoder to predict incoherent paragraphs. Non-autoregressive methods use a Transformer variant structure that removes positional encoding to extract more reliable paragraph representations to eliminate the influence of text (sentence) input order. On the other hand, existing semantic coherence restoration methods mainly focus on a single modality, thereby ignoring the impact of information integration and semantic consistency between modalities. To overcome this shortcoming, cross-modal semantic coherence restoration methods are the focus of current research.

[0004] Existing cross-modal semantic coherence restoration methods mainly introduce the annotated position information of another modality through cross-modal semantic alignment, and fuse it with the semantic information of the target modality to obtain paragraph features. Based on this, the hidden state of the Transformer decoder is initialized, decoding and encoding are performed in a non-autoregressive form, and the order of text or images is predicted using greedy search through a pointer network.

[0005] Although introducing the position information of another modality can provide effective guidance for the coherence recovery of the target modality, existing cross-modal semantic coherence recovery methods still have defects: it is very difficult to obtain data annotated with position information, which usually requires a lot of resources, so the practical application value is weak. Summary of the Invention

[0006] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide a weakly supervised cross-modal semantic coherence restoration method, which can restore cross-modal semantic coherence by using the relative position information predicted from another modality without the need for additional position information labeling.

[0007] To achieve the above-mentioned object of the invention, the weakly supervised cross-modal semantic coherence restoration method of the present invention is characterized by comprising the following steps:

[0008] (1) Obtain contextual text features and contextual image features

[0009] For the tth group of text image sets, the text set S t The input is encoded into a text encoder, which includes a BERT pre-trained model and a G-layer attention module;

[0010] First, the BERT pre-training model is used to train the text set S t N in t Unordered text Perform feature extraction to obtain text features i=1,2,…,N t , where text features d k dimensional row vector, then, the text features i=1,2,…,N t Column-by-column splicing to obtain text feature maps Feed it into the first layer of text attention block;

[0011] In the first layer of text attention block, we first get multi-head attention:

[0012] MH(Q,K,V)=[H1,H2,…,H h ]W E

[0013] Among them, [·,·] represents the concatenation of multiple attention heads, is the training parameter, each attention head H i for:

[0014]

[0015] Among them, Attention obtains attention, is the training parameter of the i-th head, d a =d k / h, Q, K, V are the query, key and value inputs of the first layer text attention block respectively, and are all text feature maps That is, Q, K,

[0016] Then, the multi-head attention MH(Q,K,V) passes through a feed-forward fully connected layer to obtain the text feature map And serve as Q, K, V input of the second layer text attention block;

[0017] The second layer of text attention block has the same structure as the first layer of text attention block, and the text feature map is obtained after the same processing. Then it is used as the Q, K, V input of the third layer text attention block, and processed in this way. The text feature map is output in the last G layer text attention block. Among them, the i-th row corresponds to the contextual text features of the i-th text, which is i=1,2,…,N t ;

[0018] At the same time, for the tth group of text image sets, the text set S t The corresponding image set X t The input is encoded into the image encoder, which includes the ViT pre-trained model and the G-layer attention module. The G-layer attention module has the same structure as the G-layer attention module in the text encoder.

[0019] First, the ViT pre-training model is used to train the image set X t M in t Unordered images Perform feature extraction to obtain image features j=1,2,…,M t , where the image features d k dimensional row vector, then, the image features j=1,2,…,M t Image feature map obtained by column splicing It is sent to the G-layer attention module and after the same processing as the G-layer attention module in the image encoder, the image feature map is obtained. The jth row corresponds to the context image features of the jth image, which is j=1,2,…,M t ;

[0020] (2) Relative position determination

[0021] First, the contextual text features i=1,2,…,N t Input with trainable parameters δ S Text relative position discriminator h δs () Modeling any two contextual text features The relative order of k≠i:

[0022]

[0023] in, is the relative position order prediction result, ρ is the feature aggregation operation;

[0024] Then, apply the logistic function to get the probability a that the i-th text comes before the k-th text t_i,k :

[0025]

[0026] In this way, we get the text position matrix A t :

[0027]

[0028] Similarly, the context image feature is With a trainable parameter δ X Image relative position discriminator h δX () Model the relative order of any two context image features and apply the logistic function to obtain the image position matrix B t :

[0029]

[0030] Among them, b t_j,l is the probability that the jth image comes before the lth image;

[0031] (3) Cross-modal semantic alignment

[0032] Use CLIP pre-trained model to train text collection S t , image set X t Perform similarity calculation to obtain the cross-modal similarity matrix Among them, the element in row i and column j is c t_ij ,i=1,2,…,N t ,j=1,2,…M t Indicates the similarity between the i-th text and the j-th image;

[0033] (4) Iterative recovery

[0034] Use an iterative matrix update algorithm to update the text position matrix A t , image position matrix B t :

[0035] Initialize a matrix A with text positions t 0 matrices of the same size Right now:

[0036]

[0037] Then determine the text position matrix A t Medium probability a t_i,k ,i=1,2,…,N t ,k=1,2,…,Nt Is it greater than the set mask threshold θ? If so, then the matrix The element in row i and column k is the probability a t_i,k , otherwise, set to 0;

[0038] The image position matrix B t Copy to matrix B′ t ,Right now:

[0039]

[0040] Then the judgment matrix element i=1,2,…,N t ,k=1,2,…,N t Is it equal to 0? If so, do not modify the image position matrix B′. t If it is not equal, then the cross-modal similarity matrix C t Find the column where the maximum value element in row i is located, and its serial number is idx t_1 , find the column where the maximum value element in row k is located, its serial number is idx t_2 , then, the image position matrix B′ t Elements in To update:

[0041]

[0042] Similarly, initialize a matrix B with the image position t 0 matrices of the same size Right now:

[0043]

[0044] Then determine the image position matrix B t Medium probability b t_j,l ,j=1,2,…,M t ,l=1,2,…,M t Is it greater than the set mask threshold θ? If so, then the matrix The element in row j and column l is the probability b t_j,l , otherwise, set to 0;

[0045] The text position matrix A t Copy as text position matrix A′ t ,Right now:

[0046]

[0047] Then the judgment matrix element j=1,2,…,M t ,l=1,2,…,M t Is it equal to 0? If so, do not modify the text position matrix A′. t If it is not equal, then the cross-modal similarity matrix C t Find the row where the maximum value element in column j is located, and its serial number is ids t_1 , find the row where the maximum value element in column l is located, its serial number is ids t_2 , then, the text position matrix A′ t Elements in To update:

[0048]

[0049] (5) Training

[0050] Using T group text set S t , image set X t Process according to steps (1) to (4), and then calculate the loss L of text semantic coherence recovery sen :

[0051]

[0052] Among them, z t_i,k is the correct order label of the i-th text and the k-th text. If the i-th text comes before the j-th text, it is 0, otherwise it is 1;

[0053] Calculate the loss L for image semantic coherence restoration img :

[0054]

[0055] Among them, y t_j,l is the correct order label of the j-th image and the l-th image. If the j-th image comes before the l-th image, it is 0, otherwise it is 1;

[0056] Based on steps (1) to (4), a weakly supervised cross-modal semantic coherence recovery network is constructed, which continuously inputs T sets of text sets S t , image set X t , according to the loss L sen , L img Train a weakly supervised cross-modal semantic coherence restoration network until convergence;

[0057] (6) Reasoning

[0058] Input the text set and the corresponding image set into the weakly supervised cross-modal semantic coherence restoration network, and obtain the text position matrix A′ according to steps (1) to (4) t , image position matrix B′ t , and then use the topological sorting algorithm to get the text order and image order respectively.

[0059] The object of the present invention is achieved in this way.

[0060] When the order of the other modality is unknown, although its absolute position information (i.e., the correct order of manual annotation) cannot be used, relative position information can be considered as a guide. The weakly supervised cross-modal semantic coherence recovery method of the present invention first trains a semantic coherence recovery model for text and image respectively, and obtains the text position matrix A t , image position matrix B t , and then with the help of the cross-modal semantic alignment capability of the CLIP pre-trained model, the relative position information of the other modality is introduced into the target modality. In order to maximize the use of cross-modal relative position information, the present invention further designs an iterative training and reasoning strategy on this basis: this method alternately freezes the text and image models during the training process, and introduces the cross-modal relative position information in the frozen model into the unfrozen model through cross-modal semantic alignment. This strategy is also applied to the reasoning stage of the model, and finally topological sorting is used to obtain the text and image sorting. Relevant experiments conducted on the SIND and TACoS-Ordering datasets demonstrate the effectiveness of the present invention, that is, cross-modal semantic coherence can be restored using the predicted relative position information of another modality without the need for additional position information annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 This is a flowchart of a specific implementation of the weakly supervised cross-modal semantic coherence restoration method of the present invention;

[0062] Figure 2 It is a schematic diagram of the principle of a specific implementation method of the weakly supervised cross-modal semantic coherence restoration method of the present invention. DETAILED DESCRIPTION

[0063] The following describes the specific embodiments of the present invention in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when detailed descriptions of known functions and designs may dilute the main content of the present invention, such descriptions will be omitted here.

[0064] Existing cross-modal semantic coherence restoration methods use a position-removed Transformer encoder to extract feature vectors of text and images during encoding, and a self-attention mechanism to extract contextual features within the paragraph. Furthermore, a cross-modal feature alignment module is used to introduce the absolute position information of the other modality into the modality to be restored, which is then fused with the contextual features to generate paragraph features. During decoding, a pointer network architecture is used as the decoder. This decoder consists of a position-removed Transformer encoder and takes as input the paragraph features and the position information of the other modality. Decoding is based on the non-autoregressive Transformer approach, with predictions obtained through greedy search.

[0065] This paper innovatively proposes a weakly supervised cross-modal semantic coherence restoration method, which can introduce the relative position information of another modality through cross-modal semantic alignment without the need for additional position information annotation, and iteratively restore the semantic coherence of the target modality.

[0066] Figure 1 、 2 They are respectively a flow chart and a principle diagram of a specific implementation method of the weakly supervised cross-modal semantic coherence restoration method of the present invention.

[0067] In this embodiment, if Figure 1 As shown, the weakly supervised cross-modal semantic coherence restoration method of the present invention includes the following steps:

[0068] Step S1: Obtain contextual text features and contextual image features

[0069] like Figure 2 As shown, for the tth group of text image sets, the text set S t The input is encoded in the text encoder, which includes the BERT pre-trained model and the G-layer attention module.

[0070] First, the BERT pre-training model is used to train the text set S t N in t Unordered text Perform feature extraction to obtain text features i=1,2,…,N t , where text features d k dimensional row vector, then, the text features i=1,2,…,N t Column-by-column splicing to obtain text feature maps Feed it into the first layer of text attention block.

[0071] Since the semantic relationship between different elements in the image or text modality plays an important role in restoring semantic coherence, this paper designs a Transformer variant structure that removes position encoding and uses a scaled dot product self-attention mechanism to utilize contextual semantic information:

[0072]

[0073] Text feature map As Q,K, The present invention adopts multiple self-attention stacks as multi-head attention applied to the text attention block of each layer.

[0074] In the first layer of text attention block, we first get multi-head attention:

[0075] MH(Q,K,v)=[H1,H2,…,H h ]W E

[0076] Among them, [·,·] represents the concatenation of multiple attention heads, is the training parameter, each attention head H i for:

[0077]

[0078] Among them, Attention obtains attention, is the training parameter of the i-th head, d a =d k / h, Q, k, V are the query, key and value inputs of the first layer text attention block respectively, and are all text feature maps That is, Q, K,

[0079] Then, the multi-head attention MH(Q,K,V) passes through a feed-forward fully connected layer to obtain the text feature map And serve as the Q, K, V input of the second layer text attention block.

[0080] The second layer of text attention block has the same structure as the first layer of text attention block, and the text feature map is obtained after the same processing. Then it is used as the Q, K, V input of the third layer text attention block, and processed in this way. The text feature map is output in the last G layer text attention block. Among them, the i-th row corresponds to the contextual text features of the i-th text, which is i=1,2,…,N t .

[0081] At the same time, for the tth group of text image sets, the text set St The corresponding image set X t The input is encoded into the image encoder, which includes the ViT pre-trained model and the G-layer attention module, where the G-layer attention module has the same structure as the G-layer attention module in the text encoder.

[0082] First, the ViT pre-training model is used to train the image set X t M in t Unordered images Perform feature extraction to obtain image features j=1,2,…,M t , where the image features d k dimensional row vector, then, the image features j=1,2,…,M t Image feature map obtained by column splicing It is sent to the G-layer attention module and after the same processing as the G-layer attention module in the image encoder, the image feature map is obtained. The jth row corresponds to the context image features of the jth image, which is j=1,2,…,M t .

[0083] In this embodiment, the BERT pre-trained model and the ViT pre-trained model, which serve as feature encoders, are preliminarily trained on the corresponding datasets. The purpose is to enable the models on the text and image sides to have preliminary semantic coherence recovery capabilities, which is an important basis for subsequent iterative training and reasoning.

[0084] Step S2: relative position determination

[0085] First, the contextual text features i=1,2,…,n t Input with trainable parameters δ I Text relative position discriminator Modeling any two contextual text features The relative order of k≠i:

[0086]

[0087] in, is the relative position order prediction result, and ρ is the feature aggregation operation, such as feature splicing.

[0088] The present invention models the pairwise relative position order prediction problem as a binary classification problem, and further applies the pairwise probability obtained by the logistic function to represent the probability of the two situations occurring. That is, the probability a that the i-th document is before the k-th document is obtained by applying the logistic function. t_i,k :

[0089]

[0090] In this way, we get the text position matrix A t :

[0091]

[0092] Similarly, the context image feature is With a trainable parameter δ X Image relative position discriminator Model the relative order of any two context image features and apply the logistic function to obtain the image position matrix B t :

[0093]

[0094] Among them, b t_j,l is the probability that the jth image comes before the lth image.

[0095] In this embodiment, the text relative position discriminator and the image relative position discriminator are preliminarily trained on the corresponding data sets. The purpose is to enable the text and image models to have preliminary semantic coherence recovery capabilities, which is an important basis for subsequent iterative training and reasoning.

[0096] Step S3: Cross-modal semantic alignment

[0097] In order to explore deeper connections between the two modalities and enhance cross-modal semantic alignment between related image and text collections, the present invention considers further utilizing cross-modal semantic similarity. t , image set x t Perform similarity calculation to obtain the cross-modal similarity matrix Among them, the element in row I and column J is C t_ij ,i=1,2,…,N t ,j=1,2,…M t Represents the similarity between the i-th text and the j-th image.

[0098] In this embodiment, the present invention fine-tunes the CLIP pre-trained model on the SIND and TACoS-ordering datasets to make it more suitable for the dataset of the present invention task. Using the fine-tuned CLIP pre-trained model, the cross-modal similarity matrix is ​​obtained. The cross-modal semantic similarity matrix is ​​used to establish a connection between the image and text modalities. This matrix enables the transfer of relative position information across modalities. Furthermore, the cross-modal relative position information guidance of the present invention is only applied to the modality to be restored if the predicted confidence level of the pairwise probability in the other modality is high.

[0099] Step S4: Iterative recovery

[0100] In order to more effectively utilize the cross-modal relative position information, this paper designs an iterative matrix update algorithm (denoted as CGO-MU) to update the position matrix of image and text modalities. For example, based on the text pair position matrix A t and the cross-modal similarity matrix C t , the CGO-MU algorithm can be used to update the image position matrix B of the image pair t B′ t =CGO-MU(B t |A t ,C t ). According to this step, the text position matrix A′ can be derived in the same way. t =CGO-MU(A t |B t ,C t ) specific update method.

[0101] Specifically, an iterative matrix update algorithm is used to update the text position matrix A t , image position matrix B t for:

[0102] Initialize a matrix A with text positions t 0 matrices of the same size Right now:

[0103]

[0104] Then determine the text position matrix A t Medium probability a t_i,k ,i=1,2,…,N t ,k=1,2,…,N t Is it greater than the set mask threshold θ? If so, then the matrix The element in row I and column k is the probability A t_i,k , otherwise, set to 0.

[0105] The image position matrix B t Copy to matrix B′ t ,Right now:

[0106]

[0107] Then the judgment matrix element i=1,2,…,N t ,k=1,2,…,N t Is it equal to 0? If so, do not modify the image position matrix B′. t If it is not equal, then the cross-modal similarity matrix C t Find the column where the maximum value element in row i is located, its serial number is idx t_1 , find the column where the maximum value element in row k is located, its serial number is idx t_2 , then, the image position matrix B′ t Elements in To update:

[0108]

[0109] Similarly, initialize a matrix B with the image position t 0 matrices of the same size Right now:

[0110]

[0111] Then determine the image position matrix B t Medium probability b t_j,l ,j=1,2,…,M t ,l=1,2,…,M t Is it greater than the set mask threshold θ? If so, then the matrix The element in row j and column l is the probability b t_j,l , otherwise, set to 0.

[0112] The text position matrix A t Copy as text position matrix A′ t ,Right now:

[0113]

[0114] Then the judgment matrix element j=1,2,…,M t ,l=1,2,…,M t Is it equal to 0? If so, do not modify the text position matrix A′. tIf it is not equal to, then the cross-modal similarity matrix C t Find the row where the maximum value element in column j is located, and its serial number is ids t_1 , find the row where the maximum value element in column l is located, and its serial number is ids t_2 , then, the text position matrix A′ t Elements in To update:

[0115]

[0116] Step S5: Training

[0117] The present invention adopts an iterative recovery mechanism in both training and inference processes, so that cross-modal relative position information can be fully utilized and promote the learning of visual and textual modal models. Based on the pre-trained text and image models, this stage of training will alternately freeze the text and image models, and through cross-modal semantic alignment, introduce the cross-modal relative position information in the frozen model into the unfrozen model. The specific optimization target is the cross-entropy loss of pairwise probability prediction. Specifically:

[0118] Using T group text set S t , image set X t Process according to steps (1) to (4), and then calculate the loss L of text semantic coherence recovery sen :

[0119]

[0120] Among them, z t_i,k is the correct order label of the i-th text and the k-th text. If the i-th text comes before the j-th text, it is 0, otherwise it is 1;

[0121] Calculate the loss L for image semantic coherence restoration img :

[0122]

[0123] Among them, y t_j,l is the correct order label of the j-th image and the l-th image. If the j-th image comes before the l-th image, it is 0, otherwise it is 1.

[0124] Based on steps (1) to (4), a weakly supervised cross-modal semantic coherence recovery network is constructed, which continuously inputs T sets of text sets S t , image set X t , according to the loss L sen 、L imgThe weakly supervised cross-modal semantic coherence restoration network is trained for R rounds until convergence. All training parameters involved in the text and image semantic coherence restoration models are updated after optimization.

[0125] Step S6: Inference

[0126] Input the text set and the corresponding image set into the weakly supervised cross-modal semantic coherence restoration network, and obtain the text position matrix A′ according to steps (1) to (4) t , image position matrix B′ t , and then use the topological sorting algorithm to get the text order and image order respectively.

[0127] Specifically, a directed graph is constructed for each set, in which each element (image or text) is regarded as a node, and the pairwise sequence prediction results represent the directed edges between the corresponding two nodes. For each node, the final order of the target set can be obtained by calculating the sum of its in / out edges and treating it as a score, and sorting the scores of all nodes to achieve the restoration of semantic coherence. The topological sorting algorithm is an existing technology and will not be further described here. The iterative recovery strategy will also be applied to the reasoning process of the model, which enables the model to perform multiple iterative updates between the two modalities, thereby improving the accuracy of semantic coherence recovery.

[0128] Analysis of experimental results:

[0129] The weakly supervised cross-modal semantic coherence recovery method proposed in the present invention can effectively encode images or texts, and iteratively restore the semantic coherence of the target modality, which is more valuable than previous methods. In order to evaluate the performance of the present invention, it is applied to two datasets, namely SIND and TACoS-Ordering. The former is a visual narrative corpus, and the latter is a dataset contributed by the present invention. The specific construction method of this dataset is as follows: According to the detailed visual and linguistic annotations of the TACoS dataset, the sentence descriptions are semantically separated by commas or periods, and there are corresponding video clips corresponding to them. On this basis, a frame of picture is randomly extracted from each clip, and an interval of at least 100 frames is left between two adjacent sampling points to ensure the visual diversity of the sampled frames.

[0130] The present invention uses perfect match rate (PMR), accuracy rate (Acc) and τ metric as evaluation indicators. Perfect match rate (PMR) measures the performance of sentence position prediction at the paragraph level. Accuracy rate (Acc) calculates the accuracy of absolute position prediction at the sentence level and is a more relaxed metric. τ metric is used to measure the relative order between all sentence pairs in the predicted paragraph, which is closer to human judgment. The present invention is compared with multiple semantic coherence recovery methods, among which LSTM+PtrNet represents the use of LSTM as an encoder and a pointer network as a decoder; L-TSort is a method that uses LSTM as an encoder and is based on topological sorting; AON optimizes the input of the decoder based on LSTM+PtrNet; RankNet is a sorting method based on Ranking; NACON is the current mainstream cross-modal semantic coherence recovery method, which uses the self-attention mechanism to extract context-integrated features in the paragraph, and introduces the correct position information marked in the other modality into the modality to be restored through the cross-modal feature alignment module.

[0131] The experimental results are shown in Table 1.

[0132]

[0133] Table 1

[0134] Table 1 shows the experimental results on the SIND and TACoS-Ordering datasets. From the experimental results, it can be seen that the performance of the method IterWeGO proposed by the present invention (Ours) on the two datasets is almost better than all existing methods, and compared with the method without iterative training and inference strategy (IterWeGO-UM), the performance of the complete IterWeGO is significantly improved, which further illustrates the effectiveness of the method of the present invention. It should be noted that although the performance of the present invention is weaker than the NACON model, NACON introduces the already labeled location information in another modality (which usually consumes a lot of resources and has low practical application value), which is naturally more conducive to semantic coherence recovery than the weak supervision method of the present invention.

[0135] Although the above describes the illustrative specific embodiments of the present invention to facilitate understanding of the present invention by those skilled in the art, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations using the concepts of the present invention are protected.

Claims

1. A weakly supervised cross-modal semantic coherence restoration method, characterized by: The following steps are involved: (1) Obtain contextual text features and contextual image features For the tth group of text image sets, the text set S t The input is encoded into a text encoder, which includes a BERT pre-trained model and a G-layer attention module; First, the BERT pre-training model is used to train the text set S t N in t Unordered text Perform feature extraction to obtain text features i=1,2,…,N t , where text features d k dimensional row vector, then, the text features i=1,2,…,N t Column-by-column splicing to obtain text feature maps Feed it into the first layer of text attention block; In the first layer of text attention block, we first get multi-head attention: MH(Q,K,V)=[H1,H2,…,H h ]W E Among them, [·,·] represents the concatenation of multiple attention heads, is the training parameter, each attention head H i for: Among them, Attention obtains attention, is the training parameter of the i-th head, d a =d k / h, Q, K, V are the query, key and value inputs of the first layer text attention block respectively, and are all text feature maps Right now Then, the multi-head attention MH(Q,K,V) passes through a feed-forward fully connected layer to obtain the text feature map And serve as Q, K, V input of the second layer text attention block; The second layer of text attention block has the same structure as the first layer of text attention block, and the text feature map is obtained after the same processing. Then it is used as the Q, K, V input of the third layer text attention block, and processed in this way. The text feature map is output in the last G layer text attention block. Among them, the i-th row corresponds to the contextual text features of the i-th text, which is i=1,2,…,N t ; At the same time, for the tth group of text image sets, the text set S t The corresponding image set X t The input is encoded into the image encoder, which includes the ViT pre-trained model and the G-layer attention module. The G-layer attention module has the same structure as the G-layer attention module in the text encoder. Use ViT pre-training model to train the image set X t M in t Unordered images Perform feature extraction to obtain image features j=1,2,…,M t , where the image features d k dimensional row vector, then, the image features j=1,2,…,M t Image feature map obtained by column splicing It is sent to the G-layer attention module and after the same processing as the G-layer attention module in the image encoder, the image feature map is obtained. The jth row corresponds to the context image features of the jth image, which is j=1,2,…,M t ; (2) Relative position determination First, the contextual text features i=1,2,…,N t Input with trainable parameters δ S Text relative position discriminator Modeling any two contextual text features The relative order of q≠i: in, is the relative position order prediction result, ρ is the feature aggregation operation; Then, apply the logistic function to get the probability a that the i-th text comes before the q-th text t_i,q : In this way, we get the text position matrix A t : Similarly, the context image feature is With a trainable parameter δ X Image relative position discriminator Model the relative order of any two context image features and apply the logistic function to obtain the image position matrix B t : Among them, b t_j,l is the probability that the jth image comes before the lth image; (3) Cross-modal semantic alignment Use CLIP pre-trained model to train text collection S t , image set X t Perform similarity calculation to obtain the cross-modal similarity matrix Among them, the element in row i and column j is c t_ij ,i=1,2,…,N t ,j=1,2,…M t Indicates the similarity between the i-th text and the j-th image; (4) Iterative recovery Use an iterative matrix update algorithm to update the text position matrix A t , image position matrix B t : Initialize a matrix A with text positions t 0 matrices of the same size Right now: Then determine the text position matrix A t Medium probability a t_i,q ,i=1,2,…,N t ,q=1,2,…,N t Is it greater than the set mask threshold θ? If so, then the matrix The element in row i and column q is the probability a t_i,q , otherwise, set to 0; The image position matrix B t Copy to matrix B′ t ,Right now: Then the judgment matrix element i=1,2,…,N t ,q=1,2,…,N t Is it equal to 0? If so, the image position matrix B is not correct. t 'Operation, if it is not equal, then in the cross-modal similarity matrix C t Find the column where the maximum value element in row i is located, and its serial number is idx t_1 , find the column where the maximum value element in row q is located, its serial number is idx t_2 , then, the image position matrix B′ t Elements in To update: Similarly, initialize a matrix B with the image position t 0 matrices of the same size Right now: Then determine the image position matrix B t Medium probability b t_j,l ,j=1,2,…,M t ,l=1,2,…,M t Is it greater than the set mask threshold θ? If so, then the matrix The element in row j and column l is the probability b t_j,l , otherwise, set to 0; The text position matrix A t Copy as text position matrix A′ t ,Right now: Then the judgment matrix element j=1,2,…,M t ,l=1,2,…,M t Is it equal to 0? If so, do not modify the text position matrix A′. t If it is not equal, then the cross-modal similarity matrix C t Find the row where the maximum value element in column j is located, and its serial number is ids t_1 , find the row where the maximum value element in column l is located, its serial number is ids t_2 , then, the text position matrix A′ t Elements in To update: (5) Training Using T group text set S t , image set X t Process according to steps (1) to (4), and then calculate the loss L of text semantic coherence recovery sen : Among them, z t_i,k is the correct order label of the i-th text and the q-th text. If the i-th text comes before the j-th text, it is 0, otherwise it is 1; Calculate the loss L for image semantic coherence restoration img : Among them, y t_j,l is the correct order label of the j-th image and the l-th image. If the j-th image comes before the l-th image, it is 0, otherwise it is 1; Based on steps (1) to (4), a weakly supervised cross-modal semantic coherence recovery network is constructed, which continuously inputs T sets of text sets S t , image set X t , according to the loss L sen , L img Train a weakly supervised cross-modal semantic coherence restoration network until convergence; (6) Reasoning Input the text set and the corresponding image set into the weakly supervised cross-modal semantic coherence restoration network, and obtain the text position matrix A′ according to steps (1) to (4) t , image position matrix B t ′, and then use the topological sorting algorithm to get the text order and image order respectively.

2. The weakly supervised cross-modal semantic coherence restoration method according to claim 1, characterized in that: The BERT pre-trained model and ViT pre-trained model as feature encoders, as well as the text relative position discriminator and image relative position discriminator, were preliminarily trained on the corresponding datasets.

Images