Multi-modal data joint embedding method based on hierarchical progressive arithmetic interaction network

Abstract

The invention discloses a multi-modal data joint embedding method based on a hierarchical progressive arithmetic interaction network, and the method comprises the following steps: obtaining text data and image data from the same semantic entity, and extracting a text local feature, a text global feature, an image local feature and an image global feature; inputting the text local feature and the image local feature into an atomic layer, and processing based on a Cartesian product to generate a first-order interaction feature; inputting the first-order interaction features into a combination layer, and carrying out nonlinear transformation processing to generate enhanced nonlinear interaction features; and inputting the nonlinear interaction features into the aggregation layer, and generating a multi-modal joint embedding vector in combination with the text global features and the image global features. The method effectively solves the problems of modal isomerism, single interaction level, lack of dynamic adaptability and the like in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal data fusion, and more particularly to a multi-modal data joint embedding method based on a hierarchical progressive arithmetic interaction network. BACKGROUND

[0002] In the digital world, the data we handle every day presents a variety of modalities, including text, images, audio, video, etc. Although these different modalities of data are unique, they often have rich associated information. How to effectively mine and utilize this associated information has become an important issue in the field of machine learning and artificial intelligence. Multi-modal embedding methods, as a key technology to address this challenge, play an important role in a variety of tasks, such as the current trend of developing towards multi-modal data in the task of question answering. Such systems not only need to process structured data, but also need to integrate unstructured data such as text, images, and sound. This multi-modal data fusion requires the system to have stronger cross-modal understanding and reasoning capabilities, and to combine different forms of data to provide accurate answers to users.

[0003] In order to achieve efficient processing of multi-modal data, joint embedding (Joint Embedding) technology has emerged, whose core idea is to map data from different modalities into a shared embedding space, so that the vectors in this space can preserve both intra-modal and inter-modal information of the original data. In other words, the goal of joint embedding is to learn a mapping function that converts data from different modalities into vector representations in the same space. Data points that are semantically similar or related should be closer in this space. Through this mapping, joint embedding not only preserves the features within each modality, but also captures the semantic associations between modalities. As a result, it can support cross-modal comparison, matching, and fusion, providing a solid technical foundation for practical engineering applications such as multi-modal intelligent question answering.

[0004] Joint embedding technology has a wide range of applications in many fields. Typical application scenarios include:

[0005] 1) Image-text matching: Given an image and a text description, joint embedding technology can determine whether the text description matches the image content, thereby enabling functions such as searching for images by text and searching for text by images.

[0006] 2) Cross-language information retrieval: In a multi-language environment, joint embedding technology can map text in different languages into the same embedding space, enabling cross-language information retrieval and semantic matching.

[0007] 3) Multimedia event detection: In the fields of video surveillance and social media, joint embedding technology can be used for detecting multimedia events such as face recognition, object tracking, and scene classification.

[0008] The implementation methods of joint embedding are diverse, among which the most representative one is the neural network-based method. The implementation process of the neural network-based joint embedding method is as follows:

[0009] 1) Data preprocessing: First, the data of different modalities need to be preprocessed, such as text segmentation, image scaling, etc., so as to input them into the neural network.

[0010] 2) Building neural network model: For data of different modalities, corresponding neural network models are built, such as convolutional neural network (CNN) for processing image data, recurrent neural network (RNN) or transformer for processing text data. These models convert the original data into vector representations.

[0011] 3) Joint embedding layer: At the end of the neural network model, a joint embedding layer is added to map the vector representations of different modalities into the same embedding space. This embedding layer is usually a fully connected layer, and its weight parameters can be obtained through training.

[0012] 4) Training and optimization: Train the neural network using labeled data, and update the model parameters through optimization algorithms (such as gradient descent) to make the vector representations of different modalities as close as possible in the embedding space to the data points that are semantically similar or related.

[0013] The limitations of the above existing methods mainly include:

[0014] 1) Lack of semantic association: In the information processing process, the features of different modalities are independently extracted and form heterogeneous spatial structures. Specifically, text features are mapped to semantic space, while image features exist in visual space. Due to this spatial heterogeneity, simply concatenating features of different modalities cannot effectively realize cross-modal semantic alignment. This processing method ignores the internal relations between local fine-grained semantic units, such as the semantic relationship between specific regions in images and related phrases in text. In this way, different modalities do not interact sufficiently in the information processing process, resulting in the inability to accurately model semantic associations. The final result is that cross-modal samples with similar semantics, such as text describing "dog" and images of dogs, are far apart in the embedding space, which is obviously not conducive to the effective integration and utilization of cross-modal information.

[0015] 2) Dimension Disaster: In the process of multi-modal data processing, the concatenation operation of high-dimensional features will cause the problem of rapid growth of input dimension. For example, the dimension of text features is 512, and the dimension of image features is 2048. After concatenation, the input dimension will be as high as 2560. Such high-dimensional input requires the model to have more parameters to achieve effective fitting. However, the significant increase in the number of parameters will further increase the risk of model overfitting, which will adversely affect the generalization ability of the model.

[0016] 3) Semantic Fault of Single Level Interaction: In existing common models such as early fusion or late fusion, the progressive interaction mechanism is generally missing. These models only perform interaction operations at a predefined level, for example, the interaction process of the visual-linguistic pre-training model ViLBERT is limited to the Transformer block level. This interaction method cannot effectively distinguish the semantic level difference between local objects and global scenes, causing the model to easily ignore fine-grained semantics when processing information. For example, in the matching process of text and image, the text phrase "dog collar" and the corresponding local area in the image fail to achieve precise alignment. This lack of semantic alignment leads to a significant increase in error rate related to details in complex question and answer tasks. For example, for the task question "What color is the dog's collar?", the model may mistakenly focus on the whole dog instead of the local area where the collar is located due to the failure to establish a hierarchical relationship between "dog collar" and the local area in the image, thus giving an incorrect answer.

[0017] 4) Bottleneck of Feature Information Transmission: Higher-order features (such as variance, mean, skewness, etc.) can better reflect global trends. However, under the single-stage interaction model architecture, the model is forced to integrate all input information at once. This integration method causes high-order features to be difficult to effectively preserve during information transmission, resulting in significant attenuation and affecting the model's accurate grasp of global features of the data.

[0018] 5) Lack of Dynamic Adaptability: In multi-modal tasks, the contribution of different modalities to the task goal changes dynamically with input data, task stage, and specific context. However, static weight allocation strategies use pre-set and fixed weight values to integrate different modal information. This rigid allocation method cannot timely and effectively capture and adapt to the dynamic fluctuations in the contribution of different modalities, thereby restricting the performance and generalization ability of the model in multi-modal tasks.

[0019] Therefore, how to solve the above technical problems, enhance the model's understanding ability, improve the model's generalization ability, and make the model perform better in complex query scenarios is a problem that needs to be solved by those skilled in the art. SUMMARY

[0020] In view of the above problems, the present application provides a multi-modal data joint embedding method based on a hierarchical progressive arithmetic interaction network to at least solve some of the technical problems mentioned in the background.

[0021] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0022] The present application provides a multi-modal data joint embedding method based on a hierarchical progressive arithmetic interaction network, comprising the following steps:

[0023] S1, obtaining text data and image data from the same semantic entity, and extracting text local features, text global features, image local features and image global features;

[0024] S2, inputting the text local features and the image local features into an atomic layer, processing based on Cartesian product, and generating first-order interaction features;

[0025] S3, inputting the first-order interaction features into a combination layer, performing nonlinear transformation processing, and generating enhanced nonlinear interaction features;

[0026] S4, inputting the nonlinear interaction features into an aggregation layer, combining the text global features and the image global features, and generating a multi-modal joint embedding vector.

[0027] Further, in the S1, the local feature vectors of the text data and the image data are projected to the same feature dimension through a linear projection layer.

[0028] Further, in the S2, in the atomic layer:

[0029] Each text local feature is paired with each image local feature to form a plurality of text-image feature pairs;

[0030] Performing basic arithmetic operations on each text-image feature pair, and concatenating the operation results along the feature dimension to form an interaction feature vector of the text-image feature pair;

[0031] The interaction feature vectors of all text-image feature pairs jointly constitute the first-order interaction features of the atomic layer.

[0032] Further, the basic arithmetic operations include element-wise addition, element-wise subtraction, element-wise multiplication and element-wise division.

[0033] Further, in the S3, in the combination layer:

[0034] Using a fully connected layer to perform dimension reduction processing on the first-order interaction features;

[0035] The first-order interaction features after dimension reduction are sequentially subjected to batch normalization processing and nonlinear activation function processing to obtain intermediate output features.

[0036] The intermediate output features are subjected to multiple nonlinear transformations, and the multiple transformation results are fused to generate enhanced nonlinear interaction features.

[0037] Further, the multiple nonlinear transformations include power transformation, logarithmic transformation and exponential transformation.

[0038] Further, in the S4, in the aggregation layer:

[0039] Obtain multiple statistics of the nonlinear interaction features in each feature dimension;

[0040] After layer normalization processing of the multiple statistics, the features are spliced along the feature dimension to form a high-order statistical feature vector;

[0041] Fuse the text global feature and the image global feature to obtain a global context feature vector;

[0042] Project the global context feature vector to a dimension matching the high-order statistical feature vector through a fully connected layer;

[0043] After element-wise addition of the high-order statistical feature vector and the projected global context feature vector, dimension reduction is performed through a fully connected layer to obtain a multi-modal joint embedding vector.

[0044] Further, the multiple statistics include mean, variance and skewness.

[0045] Further, it further comprises:

[0046] S5, the multi-modal joint embedding vector is used for a downstream task; when performing the downstream task, the similarity of the multi-modal joint embedding vector and one or more target vectors in a shared embedding space is calculated to complete the inference of the downstream task.

[0047] Further, the downstream task includes an image-text matching task, a multi-modal retrieval task or a visual question answering task.

[0048] According to the above technical solution, compared with the prior art, the present disclosure provides a multi-modal data joint embedding method based on a hierarchical progressive arithmetic interaction network, which has the following beneficial effects:

[0049] The present application designs a complete cross-modal processing architecture from input to embedding, and proposes a three-level feature interaction mechanism, effectively solving the problems of modal heterogeneity, single interaction level and lack of dynamic adaptability in the prior art.

[0050] This invention directly models the fine-grained association between text phrases and image regions through hierarchical feature extraction and atomic-level arithmetic operations, avoiding semantic alignment difficulties caused by modal heterogeneity, effectively solving the problem of missing semantic associations, and improving the detail accuracy in complex question-answering tasks.

[0051] This invention reduces feature redundancy by using fully connected layers in the combined layers (from 4Nd to d dimensions) and ReLU activation, solves the problem of a single interaction level, realizes progressive interaction from fine-grained to coarse-grained, avoids the risk of overfitting caused by high-dimensional splicing, and reduces model complexity.

[0052] This invention dynamically adapts to the contribution of different modes through statistical measures of the aggregation layer, suppresses noise, and makes the joint embedding vector robust in variable scenarios.

[0053] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0055] Figure 1 This is a schematic diagram of the multimodal data joint embedding method based on a hierarchical progressive arithmetic interaction network provided in an embodiment of the present invention.

[0056] Figure 2 This is a schematic diagram of the text feature extraction process provided in an embodiment of the present invention.

[0057] Figure 3 This is a schematic diagram of the image feature extraction process provided in an embodiment of the present invention. Detailed Implementation

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] Example 1:

[0060] This invention discloses a multimodal data joint embedding method based on a hierarchical progressive arithmetic interaction network, such as... Figure 1As shown, comprising the following steps:

[0061] S1, obtaining text data and image data from the same semantic entity, and extracting text local features, text global features, image local features and image global features;

[0062] S2, inputting the text local features and the image local features into the atomic layer, processing based on Cartesian product, and generating first-order interaction features;

[0063] S3, inputting the first-order interaction features into the combination layer, performing nonlinear transformation processing, and generating enhanced nonlinear interaction features;

[0064] S4, inputting the nonlinear interaction features into the aggregation layer, combining the text global features and the image global features, and generating a multi-modal joint embedding vector.

[0065] The multi-modal data joint embedding method based on the hierarchical progressive arithmetic interaction network provided by the application realizes unified embedding representation of multi-modal data through hierarchical processing and cross-modal progressive interaction of extracted features; the output joint embedding vector integrates different types of data together, not only enhancing the model's understanding ability of the problem, but also improving the model's generalization ability. This technology breaks through the planarization limitation of traditional joint embedding and realizes performance leap in three dimensions of semantic alignment, noise suppression and task generalization. Next, each of the above steps will be described in detail.

[0066] In the above step S1, different modal original data inputs from the same semantic entity are obtained, mainly including text data and image data; wherein the obtained text data is an original text sequence, for example, a sentence or a paragraph described in natural language; the obtained image data is original image pixel data, which can be derived from real scene shooting or synthetic images; based on the obtained text data and image data, text local features, text global features, image local features and image global features are extracted, aiming to prepare high-quality, structured feature inputs for subsequent interaction networks; specifically including:

[0067] (1) extracting local features and global features of text data:

[0068] The text data is converted into a word sequence through word segmentation processing; and BiLSTM (Bidirectional Long Short-Term Memory Network) is used to extract phrase-level text local features from the word sequence based on a preset sliding window (3-5 words), denoted as H text-phrase , and wherein N t represents the number of short sentences, d tThe feature dimension represents the local features of the text; simultaneously, a Transformer encoder is used to capture the contextual dependencies of the entire sentence from the word sequence, extracting document-level global text features, denoted as H. text-global ,and Where, d g Feature dimensions representing global text features;

[0069] like Figure 2 As shown, the original text sequence is "A black cat is sleeping on a cozy sofa". Firstly, through word segmentation, this original text sequence is transformed into a word sequence, represented as: ["A", "black", "cat", "is", "sleeping", "on", "a", "cozy", "sofa", "."]. The sliding window is set to 3, and phrase-level local text features are extracted based on a BiLSTM network, including ("A", "black", "cat"), ("black", "cat", "is")...("cozy", "sofa", "."). Each phrase is represented by a 256-dimensional vector; the generated local text features H... text-phrase The shape is [8,256]; on the other hand, the hidden state vector generated by each tag in the input sentence is average pooled, for example, the sentence "A black cat is sleeping on a cozy sofa" is encoded into a 768-dimensional vector, capturing the global semantics, and the generated text global feature H text-global The shape is

[768] ;

[0070] (2) Extracting local and global features from image data:

[0071] Faster R-CNN (Region Convolutional Neural Network) is used to identify object regions in image data and extract local image features for each object region, forming a set of local image features, denoted as H. image-region ,and Where N v d represents the number of regions. v The feature dimension represents the local features of the image; simultaneously, global image features are extracted from the image data using ResNet-50 (deep residual network), denoted as H. image-global ,and Where, d g The feature dimension representing the global features of the image is consistent with the feature dimension of the global features of the text mentioned above.

[0072] like Figure 3As shown, for the original image pixels, on the one hand, the Faster R-CNN network is used to detect salient regions in the image and extract the local image features H of each object region. image-region ;exist Figure 3 In the model, three salient object regions were detected: the cat's head, the entire cat's body, and the sofa background. Each region was represented by a 1024-dimensional vector. The extracted local image features H... image-region The shape is [3, 1024]; on the other hand, by encoding the entire image into a 2048-dimensional vector using ResNet-50, which contains the overall content and style information of the image, the generated global image feature H image-global The shape is

[2048] ;

[0073] (3) To facilitate subsequent cross-modal interaction, the local features of the text and the local features of the image need to be projected onto the same dimension d1 = 512. Dimension alignment is performed using a learnable linear projection layer (fully connected layer):

[0074] Local features of the projected text:

[0075] Local features of the projected image:

[0076] Local text features H after projection text-local The shape is [8,512], and the local features H of the projected image are... image-local The shape is [3,512];

[0077] Simultaneously, the same operation is performed on the global text features and the global image features, projecting them onto the same dimension d2 = 1024; the projected global text features H text-global The shape is

[1024] , and the local features H of the projected image are... image-global The shape is

[1024] ;

[0078] Through the above feature extraction steps, local and global features of text and image modalities were extracted respectively. Among them, the local features of text H text-local and image local features H image-local This will serve as the direct input to the atomic layers in the subsequent Hierarchical Progressive Interaction Network (HPIN); while the text global features H text-global and global features of the image H image-global It can be retained for use in the aggregation layer of the interactive network, or fused later in the model to inject global contextual information;

[0079] The hierarchical progressive interaction network (HPIN) decomposes the features of each modality into global multi-level representations to realize hierarchical feature fusion. Through progressive cross-modal interaction, multi-stage interaction modules are used to realize semantic alignment from fine granularity to coarse granularity. Another three-level feature interaction hierarchy is constructed, and feature interaction fusion is completed at the atomic layer, composition layer, and aggregation layer to capture the transition from explicit linear relationships to implicit nonlinear relationships. Then, high-order features are extracted through statistical metrics. This enables the features to gradually develop from basic representations to deep representations. The three-level structure corresponds to the atomic level (local), composition level (context), and aggregation level (global) levels of semantic granularity.

[0080] In the above step S2, the text local features and the image local features are input into the atomic layer, and first-order interaction features are generated based on Cartesian product processing.

[0081] The atomic layer is the first layer of the interaction network, and its purpose is to preliminarily explore and establish the basic association between fine-grained semantic units (phrases-regions) between modalities through basic arithmetic operations. To fully capture the interaction between different semantic units, the following operations are performed in the atomic layer:

[0082] (1) Each text local feature is paired with each image local feature based on Cartesian product to form multiple text-image feature pairs. Basic arithmetic operations are performed on each text-image feature pair. The basic arithmetic operations include element-wise addition, element-wise subtraction, element-wise multiplication, and element-wise division, which are represented as:

[0083] Add ij =h i +h j

[0084] Sub ij =h i -h j

[0085] Mul ij =h i ⊙h j

[0086]

[0087] where h i represents the i-th text local feature, h j represents the j-th image local feature, Add ij represents the operation result of element-wise addition, Sub ijdenotes the operation result obtained by element-wise subtraction; Mul ij denotes the operation result obtained by element-wise multiplication; Div ij denotes the operation result obtained by element-wise division; ∈ is used to prevent the denominator from being 0; These four operations respectively depict the inter-modal relationship from different angles: addition simulates feature fusion, subtraction highlights feature difference, multiplication strengthens co-occurring features, and division calculates feature proportion.

[0088] (2) Concatenate the operation results of each text-image feature along the feature dimension to form a 4d-dimensional interaction feature vector of the text-image feature pair;

[0089] (3) The interaction feature vectors of all text-image feature pairs together constitute the first-order interaction features of the atomic layer, which are used to capture the basic correlation between modalities; The first-order interaction features are represented as:

[0090]

[0091] wherein, Flatten is an operation function for converting high-dimensional data into a one-dimensional vector; N=N t ×N v , representing the number of all possible text-image region pairs; denotes a high-dimensional feature matrix containing a large amount of preliminary interaction information, denoted as first-order interaction features, which will serve as the input of the subsequent combination layer.

[0092] In the above step S3, the first-order interaction features are input into the combination layer for nonlinear transformation processing to generate enhanced nonlinear interaction features;

[0093] The combination layer is the second layer of the interaction network, and its purpose is to perform nonlinear transformation and compression on the basic interaction features generated by the atomic layer to capture more complex and nonlinear cross-modal dependency relationships. In the combination layer, the following operations are performed:

[0094] (1) A fully connected layer (FC Layer) is used to reduce the dimension of the first-order interaction features, compressing the dimension of the first-order interaction features from 4d back to d, to reduce redundancy and reduce computational complexity;

[0095] (2) Batch normalization (Batch Normalization, BN) is performed on the first-order interaction features after dimension reduction in turn, which is used to stabilize training and alleviate internal covariate shift; After that, the ReLU activation function introduces sparsity and filters out negative signals, thereby modeling complex dependencies to obtain intermediate output features denoted as:

[0096]

[0097] wherein, W1 represents a full connection layer weight matrix, and b1 represents a full connection layer bias, and b1 e R d ; W1 and b1 are both learnable parameters;

[0098] (3) In order to further enhance the expression ability of the model and capture complex patterns such as exponential decay and sudden growth, the intermediate output feature is subjected to multiple nonlinear transformations, and the multiple transformation results are fused to generate enhanced nonlinear interaction features;

[0099] wherein, the multiple nonlinear transformations include power transformation (Power), logarithmic transformation (Log) and exponential transformation (Exp); the power transformation amplifies significant features; the logarithmic transformation enhances the sensitivity to small changes; the exponential transformation amplifies important signals and simulates the "explosive" growth pattern; based on this, the nonlinear interaction feature is expressed as:

[0100]

[0101] wherein, the learnable parameter a in the power transformation is initialized as 2; the ∈ is added to the logarithmic transformation to prevent the logarithm from being zero; is a linear interaction feature matrix that has been subjected to nonlinear enhancement, which will be sent to the aggregation layer for high-order information extraction in the subsequent.

[0102] In the above step S4, the nonlinear interaction feature is input into the aggregation layer, and the text global feature and the image global feature are combined to generate a multi-modal joint embedding vector;

[0103] The aggregation layer is the last layer of the interaction network, and its purpose is to extract the most representative and robust high-order statistical features from the rich interaction information output by the combination layer, and fuse them with the global features extracted earlier, to finally form a joint embedding vector that can take into account both local details and global context. In the aggregation layer, the following operations are performed:

[0104] (1) The nonlinear interaction feature is regarded as a feature set containing N samples, each sample having d features; along the sample dimension (i.e. N interaction pairs), multiple statistics on each feature dimension k e {1, 2, …, d} are calculated to characterize the distribution characteristics of the interaction features; the multiple statistics include:

[0105] 1) Mean (Mean): the mean of the kth feature dimension μ k , which is used to capture the average strength of the dimension in all elements to reflect the global trend;

[0106] 2) Variance (Variance): the variance of the kth feature dimension To measure the dispersion of the characteristic dimension median, identify sensitive dimensions (high variance) and stable dimensions (low variance);

[0107] 3) Skewness (Skewness): the skewness of the kth feature dimension k , used to describe the asymmetry of the distribution, where positive skewness indicates high-value outliers, and negative skewness indicates low-value outliers;

[0108] (2) After calculating the three statistics in all d dimensions, three statistical vectors k ∈R N×d , and k ∈R N×d ; After layer normalization (LayerNorm) of the three statistics, they are spliced along the feature dimension to form a 3d-dimensional high-order statistical feature vector; denoted as:

[0109]

[0110] (3) The text global feature and the image global feature are fused through a simple summation smoothing operation to obtain a unified global context feature vector; then, the global context feature vector is projected to a dimension matching the high-order statistical feature vector through a fully connected layer;

[0111]

[0112] (4) After element-wise addition of the high-order statistical feature vector and the projected global context feature vector , a fully connected layer is used for dimension reduction to obtain a multi-modal joint embedding vector denoted as:

[0113]

[0114] The global context feature vector integrates cross-modal interaction information from fine-grained to coarse-grained, which can be used for downstream tasks such as image-text matching, multi-modal retrieval, visual question answering, etc. By calculating the similarity (such as cosine similarity) between the embedding vector and the target vector in the shared embedding space, the prediction task can be completed.

[0115] Embodiment two:

[0116] The embodiment two of the present application further comprises:

[0117] S5, using the multi-modal joint embedding vector for a downstream task; when performing the downstream task, completing the inference of the downstream task by calculating the similarity between the multi-modal joint embedding vector and one or more target vectors in the shared embedding space;

[0118] When the downstream task is image-text matching, the implementation is that a multi-modal joint embedding vector is generated for an image-text pair to be matched, and whether the image and the text match is predicted by calculating the similarity between the multi-modal joint embedding vector and a learnable matching threshold vector in the shared embedding space.

[0119] When the downstream task is multi-modal retrieval, the implementation is that a multi-modal joint embedding vector is generated for a query image-text pair, denoted as a query vector; meanwhile, multi-modal joint embedding vectors are respectively generated for a plurality of target image-text pairs in a candidate library, denoted as candidate vectors; the most relevant retrieval result is predicted and returned by calculating the similarity between the query vector and all the candidate vectors in the shared embedding space and ranking them in order.

[0120] When the downstream task is visual question answering, the implementation is that a given image and question are input as a whole to generate a multi-modal joint embedding vector, denoted as a question vector; meanwhile, each answer in a predefined candidate answer set is regarded as a text and paired with an empty image or a question image to generate a corresponding multi-modal joint embedding vector, denoted as a candidate vector; the most possible answer is predicted by calculating the similarity between the question vector and all the candidate vectors.

[0121] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0122] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multimodal data joint embedding method based on hierarchical progressive arithmetic interaction networks, characterized in that, Includes the following steps: S1. Obtain text data and image data from the same semantic entity, and extract local text features, global text features, local image features, and global image features; S2. Input the local text features and the local image features into the atomic layer, process them based on the Cartesian product, and generate first-order interactive features; S3. Input the first-order interaction features into the combination layer and perform nonlinear transformation processing to generate enhanced nonlinear interaction features. S4. Input the nonlinear interaction features into the aggregation layer, and combine the text global features and the image global features to generate a multimodal joint embedding vector.

2. The multimodal data joint embedding method based on hierarchical progressive arithmetic interaction networks according to claim 1, characterized in that, S1 further includes: projecting the local feature vectors of the text data and the image data to the same feature dimension through a linear projection layer.

3. The multimodal data joint embedding method based on hierarchical progressive arithmetic interaction networks according to claim 1, characterized in that, In S2, within the atomic layer: Each text local feature is paired with each image local feature to form multiple text-image feature pairs; Perform basic arithmetic operations on each text-image feature pair and concatenate the results along the feature dimension to form the interactive feature vector of that text-image feature pair; The interaction feature vectors of all text-image feature pairs are collectively used to form the first-order interaction features of the atomic layer.

4. The multimodal data joint embedding method based on hierarchical progressive arithmetic interaction networks according to claim 3, characterized in that, The basic arithmetic operations include: element-wise addition, element-wise subtraction, element-wise multiplication, and element-wise division.

5. The multimodal data joint embedding method based on hierarchical progressive arithmetic interaction networks according to claim 1, characterized in that, In S3, in the combined layer: The first-order interaction features are reduced in dimensionality using a fully connected layer. The first-order interactive features after dimensionality reduction are subjected to batch normalization and nonlinear activation function processing to obtain intermediate output features. Multiple nonlinear transformations are applied to the intermediate output features, and the results of these transformations are fused together to generate enhanced nonlinear interactive features.

6. The multimodal data joint embedding method based on hierarchical progressive arithmetic interaction networks according to claim 5, characterized in that, The various nonlinear transformations include power transformation, logarithmic transformation, and exponential transformation.

7. The multimodal data joint embedding method based on hierarchical progressive arithmetic interaction networks according to claim 1, characterized in that, In S4, in the polymer layer: Obtain multiple statistics for the nonlinear interaction features in each feature dimension; After performing layer normalization on the multiple statistics, they are concatenated along the feature dimension to form a high-order statistical feature vector. The global text features and the global image features are fused to obtain a global context feature vector; The global context feature vector is projected onto a dimension that matches the higher-order statistical feature vector through a fully connected layer; After element-wise addition of the higher-order statistical feature vector and the projected global context feature vector, the dimensionality is reduced through a fully connected layer to obtain the multimodal joint embedding vector.

8. The multimodal data joint embedding method based on hierarchical progressive arithmetic interaction networks according to claim 7, characterized in that, The statistics include mean, variance, and skewness.

9. The multimodal data joint embedding method based on hierarchical progressive arithmetic interaction networks according to claim 1, characterized in that, Also includes: S5. Use the multimodal joint embedding vector for downstream tasks; when executing the downstream tasks, calculate the similarity between the multimodal joint embedding vector and one or more target vectors in the shared embedding space to complete the inference of the downstream tasks.

10. The multimodal data joint embedding method based on hierarchical progressive arithmetic interaction networks according to claim 9, characterized in that, The downstream tasks include image-text matching tasks, multimodal retrieval tasks, or visual question answering tasks.

Images