Cross-modal semantic alignment method based on visual semantic shared space

Abstract

The invention relates to a cross-modal semantic alignment method based on a visual semantic shared space. The method comprises the following steps: S1, acquiring an image embedding representation and a text embedding representation; s2, inputting the embedded representations into a routing gate module, and determining a routing probability value of each embedded representation; s3, inputting the embedded representation into an aggregation expert module, and aggregating the embedded representation into an aggregation vector result by the obtained overall vector; and S4, a loss function is calculated based on an aggregation vector result, a routing gate module and an aggregation expert module are trained based on the loss function, a cross-modal semantic alignment model is obtained, and the cross-modal semantic alignment model is used for realizing cross-modal semantic alignment of actual image data and text data in a multi-modal task. Compared with the prior art, the cross-modal semantic alignment method has the advantages of improving the cross-modal semantic alignment performance and the like.

Description

Technical Field

[0001] The present invention relates to the field of cross-modal semantic alignment, and in particular to a cross-modal semantic alignment method based on a visual semantic shared space. Background Art

[0002] For current artificial intelligence, using natural language to identify and describe the visual world is a necessary capability, which has promoted the research on establishing accurate and generalized semantic alignment between visual and textual data. Constructing a common embedding space for vision and text is a typical way to achieve semantic alignment between cross-modal data. The reason is that it does not require additional cross-modal interaction and the implementation principle is simple. After mapping the extracted visual fragment embedding and text fragment embedding to the same semantic space, they are aggregated into a vector of fixed length, and the distance between visual and textual instances in the space is measured by similarity calculation. The goal is to optimize the distance between paired instances to the minimum. Therefore, it is highly efficient in practical applications. It has been widely used in various multimodal tasks such as image description generation, visual question answering, image text matching, and zero-shot image recognition. Therefore, research on this issue is a current hot topic.

[0003] Wang et al. proposed a dual-branch neural network with maximum margin sorting loss and new neighborhood constraints to construct a semantic embedding space to learn the correspondence between image and text data. Wehrmann et al. proposed a character-based alignment and embedding method for language encoders. Based on this, Song et al. proposed a method to optimize multimodal embedding by considering the polysemy information of a word within the modality, while Wang et al. focused on the consistency information within the modality to improve the semantic embedding space. Recently, Chun et al. sampled instances and achieved further improvements as probability distributions, while Li et al. optimized the image-text embedding through visual and textual semantic reasoning to improve the semantic alignment process. Considering that the key to cross-modal semantic alignment based on the shared space of visual semantics is to learn powerful image and text embeddings to obtain a high-quality joint embedding space, many current studies focus on the feature aggregation of visual semantic embeddings. The most commonly used aggregation method is a simple pooling function. For example, average pooling and maximum pooling are used to detect the average and salient points of features. There are also studies that choose to combine different pooling functions for feature aggregation. Some researchers have achieved feature aggregation by designing complex aggregators, which also introduces a large number of parameters into the training process, such as aggregation methods based on cross-modal attention and self-attention mechanisms. Zhang et al. achieved cross-modal guided feature aggregation by dynamically focusing on local information. These studies usually take more time than simple aggregation methods, and the results are not superior. Therefore, Chen et al. and Zhang et al. are committed to designing adaptive aggregation strategies to learn the best combination of simple pooling methods, and have achieved competitive results. Most of the current research focuses on how to aggregate fragment features into fixed-length embedding vectors and proposes some complex feature aggregators, such as those based on self-attention mechanisms or intra-modal attention mechanisms, and uses exactly the same feature aggregator for the entire dataset, but they have not noticed the problem of semantic differences in data subsets.

[0004] Specifically, different data have different intra-modal relationships, that is, the number of entities in the modality and the relationships between them are different, which leads to the existence of potential data subsets or different topics due to semantic differences between data. The feature aggregation module in the prior art, which is trained uniformly based on the entire data set, has errors and noise in the overall embedding vectors learned for different potential data subsets, which in turn affects the performance of cross-modal semantic alignment. Summary of the invention

[0005] The purpose of the present invention is to provide a cross-modal semantic alignment method based on visual semantic shared space in order to reduce the noise of feature aggregation in feature alignment tasks and thereby improve the performance of cross-modal semantic alignment.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A cross-modal semantic alignment method based on visual semantic shared space, the method comprising:

[0008] S1, obtain image g and text t, input image g and text t into the corresponding modal encoder for encoding to obtain image embedding representation G and text embedding representation T;

[0009] S2. Input the embedded representation into the routing gate module, calculate the query vector and key vector corresponding to the embedded representation, perform dot multiplication on the query vector and the key vector after passing through the fully connected layer, and perform noise reduction on the result through the gate mask module, and calculate the value vector after adding the gate attention to obtain the attn feature, use the fully connected layer on the attn feature, and convert it into the routing gate representation vector Z. Based on the routing gate representation vector Z, determine the routing probability value of each embedded representation, and determine the routing probability of the optimal pooling expert. The routing probability value means the probability that the embedded representation is routed to a pooling expert.

[0010] S3, inputting the embedded representation into an aggregation expert module, the aggregation expert module is composed of multiple pooling experts, the selected pooling expert aggregates the embedded representation into an aggregated vector result based on the routing probability value and the overall vector obtained by the weighted sum of the features, and the selected pooling expert is the optimal pooling expert;

[0011] S4. Calculate the loss function based on the aggregation vector result, train the routing gate module and the aggregation expert module based on the loss function, and obtain a cross-modal semantic alignment model. The cross-modal semantic alignment model is used to control the actual image data and text data to achieve cross-modal semantic alignment in multimodal tasks.

[0012] Furthermore, the specific steps of S2 are:

[0013] The embedded representation is input into the routing gate module, and the query vector and key vector corresponding to the embedded representation are calculated. The query vector and the key vector are point-multiplied after passing through the fully connected layer, and the result F is passed through the gate mask module. In the gate mask module, the gate mask of the query vector and the key vector is calculated, and the gate mask is used to reduce the noise of the query vector and the key vector, and the value vector attn feature after adding attention is calculated. The attn feature is converted into the routing gate representation vector Z using a layer of full connection. The routing gate representation vector Z and the embedded representation are concatenated and then pass through the fully connected layer to generate the routing probability value;

[0014] The process of calculating the gate mask of the query vector and the key vector can be:

[0015]

[0016] in, is the result of element-wise multiplication of the query and key vectors, the gate mask M of the query and key vectors Q , M K Generated by two fully connected layers and sigmoid activation function, where σ represents the sigmoid operation, Expression weight, Describe the bias term.

[0017] Furthermore, the calculation process of the routing gate representation vector Z is expressed as:

[0018]

[0019] Z=attnW attn +b attn

[0020] where the softmax function is applied to each row, represents the weight, represents the bias term, and m represents the number of pooling experts.

[0021] Furthermore, the routing probability value is:

[0022]

[0023]

[0024] where p i (G) represents the routing probability value of the image embedding representation for the i-th pooling expert, p i (T) represents the routing probability value of the text embedding representation for the i-th pooling expert;

[0025] The routing probability of the optimal pooling expert for the image embedding representation G is p(G)=max{p i (G)} is calculated; for the text embedding representation T = {t1, t2, ..., t s}, the routing probability calculation expression of the best pooling expert of T is p(T)=max{p i (T)}, Z represents the routing gate representation vector.

[0026] Furthermore, the specific steps of S3 are:

[0027] The aggregation expert module sorts the embedding representations on the feature dimension to be aggregated, selects the embedding representations corresponding to the first k feature values, and assigns corresponding weights to each of the first k embedding representations based on the aggregation expert module calculation mechanism of the corresponding selected pooling expert, calculates the weight and the weighted sum of the first k embeddings, obtains the overall vector, multiplies the overall vector and the routing probability value, and obtains the aggregation vector result.

[0028] Furthermore, the aggregation expert module calculation mechanism is implemented based on an encoder-decoder structure.

[0029] Furthermore, the encoder in the encoder-decoder structure is a position encoder, and the position encoder encodes the region positions of the first k embedded representations to obtain corresponding position vectors.

[0030] Furthermore, the decoder in the encoder-decoder structure is a position decoder, which decodes the position vector to obtain weights corresponding to the first k embedded representations, and the decoder is a sequence model based on BiGRU that can generate pooling coefficients.

[0031] Furthermore, the loss function is specifically the sum of a bidirectional ternary sorting loss function and a load balancing loss function.

[0032] Furthermore, the load balancing loss function is:

[0033]

[0034] Where S i represents the proportion of samples assigned to pooling expert i, P i represents the proportion of routing probability assigned to pooling expert i.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] The present invention proposes a combination of a routing gate module and an aggregation expert module, and further combines multiple aggregation operators to aggregate features for different data subsets. A gate mask module is set in the routing gate module to effectively achieve feature denoising, while the aggregation expert module combines multiple pooling experts to pool and aggregate fragment features of different samples, and an additional auxiliary load balancing objective function can promote load balancing between pooling experts. The present invention effectively solves the problem of semantic differences in data subsets, improves the feature aggregation process in a shared space, and improves the semantic alignment effect between cross-modal features. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 The data flow diagram of the routing gate module of the present invention;

[0038] Figure 2It is a data flow chart of the aggregation expert module of the present invention;

[0039] Figure 3 It is a schematic diagram of the load balancing loss function of the present invention;

[0040] Figure 4 It is the overall flow chart of the present invention. DETAILED DESCRIPTION

[0041] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0042] The purpose of the present invention is to provide a cross-modal semantic alignment method based on the idea of ​​hybrid experts. Based on the field of deep learning, a hybrid pooling expert framework is proposed. The framework will combine multiple aggregation operators to aggregate features for different data subsets, effectively solve the problem of semantic differences in data subsets, improve the feature aggregation process in the shared space, and improve the semantic alignment effect between cross-modal features.

[0043] The present invention proposes a semantic alignment method based on the visual semantic space paradigm, which establishes a hybrid pooling expert framework through a routing gate module, an aggregation expert module and a loss function module. This method can be used for multimodal downstream tasks such as image-text matching tasks to achieve cross-modal semantic feature alignment and improve task performance.

[0044] The method of the present invention comprises the following steps:

[0045] S1, obtain image g and text t, input image g and text t into the corresponding modal encoder for encoding to obtain image embedding representation G and text embedding representation T;

[0046] S2. Input the embedded representation into the routing gate module, calculate the query vector and key vector corresponding to the embedded representation, perform dot multiplication on the query vector and the key vector after passing through the fully connected layer, and denoise the result through the gate mask module, and calculate the value vector after adding the gate attention to obtain the attn feature, use the fully connected layer on the attn feature, and convert it into the routing gate representation vector Z. Based on the routing gate representation vector Z, determine the routing probability value of each embedded representation, and the routing probability value means the probability that the embedded representation is routed to a pooling expert;

[0047] S3, inputting the embedded representation into an aggregation expert module, wherein the aggregation expert module is composed of multiple pooling experts, and the selected pooling expert aggregates the embedded representation into an aggregated vector result based on the routing probability value and the overall vector obtained by the weighted sum of the features;

[0048] S4. Calculate the loss function based on the aggregation vector result, train the routing gate module and the aggregation expert module based on the loss function, and obtain a cross-modal semantic alignment model. The cross-modal semantic alignment model is used to realize cross-modal semantic alignment of actual image data and text data in multimodal tasks.

[0049] In S4, the cross-modal semantic alignment model of the present invention is used to control the actual image data and text data to achieve cross-modal semantic alignment in a multimodal task. In the actual multimodal task, the unaligned image data and text data are first obtained, and the unaligned image data and text data are input into the cross-modal semantic alignment model. The cross-modal semantic alignment model outputs the actual aggregation vector result, which is the semantically aligned data.

[0050] The purpose of S1 is multimodal feature encoding. Assuming that g represents the input image, t represents the input text, and (g, t) represents an image-text pair, it can be encoded based on different modal encoders to obtain the corresponding embedding representation, where n represents the number of extracted image segments (depending on the visual encoder, it can be divided into region, grid, and patch), s represents the number of words extracted from the text, and d represents the dimension of the feature.

[0051] In S2, for the input image data and text data, this method uses the routing gate module to make full use of the self-attention mechanism based on the intra-modal relationship to route each sample to the appropriate pooling expert for feature aggregation. It is worth noting that only the detailed routing process of the visual branch will be described here, because the routing principle of the text branch is the same. Figure 1 The algorithm structure diagram of this module is as follows. First, a set of regional features G = {g1, g2, ..., g n} as the input of the routing gate strategy, where make Represent the query, key and value vectors in the attention mechanism, where n represents the number of regions and d refers to the dimension of the vector. Considering that the dot product between the query and the key may contain noise, this module proposes a gate mechanism to filter out useless information and retain useful information. First, the module needs to calculate the gate mask of the query and key vectors. After that, the model uses the gate mask to reduce the noise of the query and key vectors, and calculates the value vector attn after adding attention. Then, the attn feature with added gate attention is converted into a routing gate representation vector Z using a layer of full connection. Finally, each image G and each text T is routed to the pooling expert of the corresponding modality with the highest probability based on the normalized distribution on the m pooling experts, which can also be called an aggregation operator.

[0052] In the routing gate module, considering that the dot product between the query and the key may contain noise, this module proposes a gate mechanism to filter out useless information and retain useful information. First, the module needs to calculate the gate mask of the query and key vectors. The calculation process can be expressed as:

[0053]

[0054] in is the result of element-wise multiplication of the query and key vectors. Afterwards, the gate mask M of Q and K Q , M K Generated by two fully connected layers and sigmoid activation function, where σ represents the sigmoid operation,

[0055] After that, the model uses the gate mask to reduce the noise of the query and key vectors, and calculates the value vector attn after adding attention. Then, the attn feature with added gate attention is converted into the routing gate representation vector Z using a layer of full connection. The calculation process can be expressed as:

[0056]

[0057] Z=attnW attn +b attn

[0058] where the softmax function is applied to each row, m represents the number of pooling experts.

[0059] Finally, each image G and each text T is routed to the pooling expert of the corresponding modality with the highest probability based on the normalized distribution on the m pooling experts, which can also be called an aggregation operator. The formula process for calculating the probability of G and T being routed to expert i can be expressed as:

[0060]

[0061]

[0062] The routing probability of the optimal pooling expert about G can be expressed as p(G)=max{p i (G)} is calculated. For word-level text features T = {t1, t2, ..., t s}, the optimal pooling expert routing probability calculation expression for sentence T is p(T) = max{p i (T)}.

[0063] In S3, in order to aggregate the visual and text fragment features in the visual semantic shared space into holistic vector embeddings respectively, thereby measuring the similarity of the visual and text modalities, this method designs a special aggregation expert module. This module consists of m pooling experts, each of which pools and aggregates the routed images and texts based on the same model structure. Specifically, this module first sorts the unimodal fragment features on the feature dimension to be aggregated, then assigns weights to the sorted features based on a certain mechanism, and calculates the weighted sum between the features to obtain the corresponding overall vector, and finally multiplies it with the probability value calculated by the routing gate module to obtain the final aggregation vector result. Figure 2 This is the algorithm structure diagram of the aggregation expert module. For a set of inputs with n regional features The selected pooling expert aggregates it into a fixed-length vector Let max k (·) represents the function of extracting the first k values ​​from the feature sorting list, and the coefficient θ k It is the corresponding weight of the kth highest value after sorting the feature value on the regional feature dimension to be aggregated, and 1≤k≤n. The weight value is the normalized distribution on n regions. p(G) represents the routing probability value of the image G calculated from the routing gate module. Pooling experts need to obtain θ based on a certain calculation mechanism. k The paper uses an encoder-decoder structure to achieve this goal. Specifically, the structure consists of two parts: the position encoder, a position encoding function based on trigonometric functions, which can encode the regional position of the feature to obtain the corresponding vectorized representation; the position decoder, a sequence model based on BiGRU that can generate pooling coefficients, which can process the vector obtained by the encoder.

[0064] In S4, in order to constrain the process of cross-modal semantic alignment, that is, to keep higher similarity scores between semantically similar samples and lower scores between semantically irrelevant samples, this paper designs a special loss function module, which includes two loss functions. One is the bidirectional ternary ranking loss function, which can optimize the distance between cross-modal semantic features; the other is the load balancing loss function, which can optimize the balanced load of the samples routed to each pooling expert. Figure 3 Schematic diagram of the load balancing loss function.

[0065] The bidirectional ternary sorting loss function of the present invention is a semantic alignment objective function, and the load balancing loss function can achieve balanced load on routing experts. For a given m experts and batches, The formula of the load balancing loss function can be expressed as:

[0066]

[0067] Where S i represents the proportion of samples assigned to expert i, P i represents the proportion of routing probability assigned to expert i.

[0068] The routing gate module can adaptively assign each sample to the appropriate pooling operator based on the different intra-modal relationships, while the aggregation expert module combines multiple pooling experts to pool and aggregate the fragment features of different samples, and the additional auxiliary load balancing objective function can promote load balancing among pooling experts. This framework is based on the visual semantic shared space paradigm and is applicable to a variety of image encoders and text encoders. It can be plug-and-play in different multimodal related tasks to promote cross-modal semantic alignment. The overall framework of this method is shown in the figure below. Figure 4 shown.

[0069] The references mentioned in the present invention are as follows:

[0070] Faghri F, Fleet DJ, Kiros JR, et al. Vse++: Improving visual-semanticembeddings with hard negatives[C] / / Proceedings of the British Machine Vision Conference (BMVC). 2018:1-13.

[0071] Frome A,Corrado GS,Shlens J,et al.Devise:A deep visual-semanticembedding model[J].Advances in neural information processing systems,2013,26.

[0072] Kiros R,Salakhutdinov R,Zemel R S.Unifying visual-semantic embeddingswith multimodal neural language models[C] / / Advances in Neural InformationProcessing Systems.2014:210-218.

[0073] Karpathy A,Fei-Fei L.Deep visual-semantic alignments for generatingimage descriptions[C] / / Proceedings of the IEEE conference on computer visionand pattern recognition.2015:3128-3137.

[0074] Fan C,Zhang X,Zhang S,Wang W,Zhang C,Huang H.Heterogeneous memoryenhanced multimodal attention model for video question answering[C] / / Proceedings of CVPR.2019:1999-2007.

[0075] Li K,Zhang Y,Li K,et al.Visual semantic reasoning for image-textmatching[C] / / Proceedings of the IEEE / CVF international conference on computervision.2019:4654-4662.

[0076] Changpinyo S,Chao W L,Gong B,et al.Synthesized classifiers for zero-shot learning[C] / / Proceedings of the IEEE conference on computer vision andpattern recognition.2016:5327-5336.

[0077] Wang L,Li Y,Huang J,et al.Learning two-branch neural networks forimage-text matching tasks[J].IEEE Transactions on Pattern Analysis andMachine Intelligence,2018,41(2):394-407.

[0078] Wehrmann J,Souza D M,Lopes M A,et al.Language-agnostic visual-semantic embeddings[C] / / Proceedings of the IEEE / CVF International Conferenceon Computer Vision.2019:5804-5813.

[0079] Song Y,Soleymani M.Polysemous visual-semantic embedding for cross-modal retrieval[C] / / Proceedings of the IEEE / CVF Conference on Computer Visionand Pattern Recognition.2019:1979-1988.

[0080] Wang H,Zhang Y,Ji Z,et al.Consensus-aware visual-semantic embeddingfor image-text matching[C] / / Computer Vision–ECCV 2020:16th EuropeanConference,Glasgow,UK,August 23–28,2020,Proceedings,Part XXIV 16.SpringerInternational Publishing,2020:18-34.

[0081] Chun S,Oh S J,De Rezende R S,et al.Probabilistic embeddings forcross-modal retrieval[C] / / Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition.2021:8415-8424.

[0082] Li K,Zhang Y,Li K,et al.Image-text embedding learning via visual andtextual semantic reasoning[J].IEEE transactions on pattern analysis andmachine intelligence,2022,45(1):641-656.

[0083] Qu L,Liu M,Cao D,et al.Context-aware multi-view summarization networkfor image-text matching[C] / / Proceedings of the 28th ACM InternationalConference on Multimedia.2020:1047-1055.

[0084] Zhang Y,Lu H.Deep cross-modal projection learning for image-textmatching[C] / / Proceedings of the European conference on computer vision(ECCV).2018:686-701.

[0085] Li J,Liu L,Niu L,et al.Memorize,associate and match:Embeddingenhancement via fine-grained alignment for image-text retrieval[J].IEEETransactions on Image Processing,2021,30:9193-9207.

[0086] Lee K H,Chen X,Hua G,et al.Stacked cross attention for image-textmatching[C] / / Proceedings of the European conference on computer vision(ECCV).2018:201-216.

[0087] Wehrmann J,Kolling C,Barros R C.Adaptive cross-modal embeddings forimage-text alignment[C] / / Proceedings of the AAAI conference on artificialintelligence.2020,34(07):12313-12320.

[0088] Han N,Chen J,Xiao G,et al.Fine-grained cross-modal alignment networkfor text-video retrieval[C] / / Proceedings of the 29th ACM InternationalConference on Multimedia.2021:3826-3834.

[0089] Zhang G,Wei S,Pang H,et al.Heterogeneous feature fusion and cross-modal alignment for composed image retrieval[C] / / Proceedings of the 29th ACMinternational conference on multimedia.2021:5353-5362.

[0090] Chen J,Hu H,Wu H,et al.Learning the best pooling strategy for visualsemantic embedding[C] / / Proceedings of the IEEE / CVF conference on computervision and pattern recognition.2021:15789-15798.

[0091] Zhang Z,Shu C,Xiao Y,et al.Improving visual-semantic embedding with adaptive pooling and optimization objective[C] / / Proceedings of the 17thConference of the European Chapter of the Association for ComputationalLinguistics.2023:1217-1229

[0092] Jacobs RA,Jordan MI,Nowlan SJ,et al.Adaptive mixtures of localexperts[J].Neural computation,1991,3(1):79-87.

[0093] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A cross-modal semantic alignment method based on visual semantic shared space, characterized in that the method include: S1, obtain image g and text t, input image g and text t into the corresponding modal encoder for encoding to obtain image embedding representation G and text embedding representation T; S2. Input the embedded representation into the routing gate module, calculate the query vector and key vector corresponding to the embedded representation, perform dot multiplication on the query vector and the key vector after passing through the fully connected layer, and perform noise reduction on the result through the gate mask module, and calculate the value vector after adding the gate attention to obtain the attn feature, use the fully connected layer on the attn feature, and convert it into the routing gate representation vector Z. Based on the routing gate representation vector Z, determine the routing probability value of each embedded representation, and determine the routing probability of the optimal pooling expert. The routing probability value means the probability that the embedded representation is routed to a pooling expert. S3, inputting the embedded representation into an aggregation expert module, the aggregation expert module is composed of multiple pooling experts, the selected pooling expert aggregates the embedded representation into an aggregated vector result based on the routing probability value and the overall vector obtained by the weighted sum of the features, and the selected pooling expert is the optimal pooling expert; S4. Calculate the loss function based on the aggregation vector result, train the routing gate module and the aggregation expert module based on the loss function, and obtain a cross-modal semantic alignment model. The cross-modal semantic alignment model is used to control the actual image data and text data to achieve cross-modal semantic alignment in multimodal tasks.

2. According to claim 1, a cross-modal semantic alignment method based on visual semantic shared space is characterized in that: The specific steps of S2 are: The embedded representation is input into the routing gate module, and the query vector and key vector corresponding to the embedded representation are calculated. The query vector and the key vector are point-multiplied after passing through the fully connected layer, and the result F is passed through the gate mask module. In the gate mask module, the gate mask of the query vector and the key vector is calculated, and the gate mask is used to reduce the noise of the query vector and the key vector, and the value vector attn feature after adding attention is calculated. The attn feature is converted into the routing gate representation vector Z using a layer of full connection. The routing gate representation vector Z and the embedded representation are concatenated and then pass through the fully connected layer to generate the routing probability value; The process of calculating the gate mask of the query vector and the key vector can be: in, is the result of element-wise multiplication of the query and key vectors, the gate mask M of the query and key vectors Q , M K Generated by two fully connected layers and sigmoid activation function, where σ represents the sigmoid operation, Expression weight, Describe the bias term.

3. According to claim 2, a cross-modal semantic alignment method based on visual semantic shared space is characterized in that: The calculation process of the routing gate representation vector Z is expressed as: Z=attnW attn +b attn where the softmax function is applied to each row, represents the weight, represents the bias term, and m represents the number of pooling experts.

4. According to claim 3, a cross-modal semantic alignment method based on visual semantic shared space is characterized in that: The routing probability value is: where p i (g) represents the routing probability value of the image embedding representation for the i-th pooling expert, p i (T) represents the routing probability value of the text embedding representation for the i-th pooling expert; The routing probability of the optimal pooling expert for the image embedding representation G is p(G)=max{p i (G)} is calculated; for the text embedding representation T = {t1, t2, ..., t s }, the routing probability calculation expression of the best pooling expert of T is p(T)=max{p i (T)}, Z represents the routing gate representation vector.

5. According to the cross-modal semantic alignment method based on visual semantic shared space according to claim 1, it is characterized in that: The specific steps of S3 are: The aggregation expert module sorts the embedding representations on the feature dimension to be aggregated, selects the embedding representations corresponding to the first k feature values, and assigns corresponding weights to each of the first k embedding representations based on the aggregation expert module calculation mechanism of the corresponding selected pooling expert, calculates the weight and the weighted sum of the first k embeddings, obtains the overall vector, multiplies the overall vector and the routing probability value, and obtains the aggregation vector result.

6. The cross-modal semantic alignment method based on visual semantic shared space according to claim 5, characterized in that: The aggregation expert module calculation mechanism is implemented based on an encoder-decoder structure.

7. The cross-modal semantic alignment method based on visual semantic shared space according to claim 6, characterized in that: The encoder in the encoder-decoder structure is a position encoder, which encodes the region positions of the first k embedded representations to obtain corresponding position vectors.

8. The cross-modal semantic alignment method based on visual semantic shared space according to claim 7, characterized in that: The decoder in the encoder-decoder structure is a position decoder, which decodes the position vector to obtain weights corresponding to the first k embedded representations, and the decoder is a sequence model based on BiGRU that can generate pooling coefficients.

9. The cross-modal semantic alignment method based on visual semantic shared space according to claim 1, characterized in that: The loss function is specifically the sum of a bidirectional ternary sorting loss function and a load balancing loss function.

10. The cross-modal semantic alignment method based on visual semantic shared space according to claim 9, characterized in that: The load balancing loss function is: Where S i represents the proportion of samples assigned to pooling expert i, P i represents the proportion of routing probability assigned to pooling expert i.

Images