Multi-modal feature alignment method based on implicit feature space

Abstract

The invention relates to a multi-modal feature alignment method based on an implicit feature space, and belongs to the field of artificial intelligence and multi-modal data processing. According to the method, a text is taken as a center, and a pre-trained large model is used for carrying out feature extraction and fine adjustment on image and audio information. Secondly, constructing an implicit feature space capable of capturing deep association between different data types, and generating feature representation reflecting the internal relation of each modal by utilizing a contrast learning framework fine tuning model; compared with a traditional method, the technology does not depend on explicit corresponding annotation, the requirement for a large-scale annotation data set is reduced, and therefore the generalization ability and adaptability of the model are improved.

Description

Technical Field

[0001] The present invention belongs to the fields of artificial intelligence and multimodal data processing, and particularly relates to a multimodal feature alignment method based on implicit feature space. Background Art

[0002] As the complexity and diversity of new media content continue to grow, traditional approaches to processing multimodal data have encountered numerous limitations. These approaches typically focus on a single type of data and fail to fully integrate the complex relationships between multiple information sources, such as text, images, and audio. This limits their effectiveness in complex scenarios.

[0003] To address these issues, this paper proposes a multimodal feature alignment method based on an implicit feature space. This method focuses on text and uses a large, pre-trained model to extract and fine-tune features from images and audio. It then constructs an implicit feature space that captures the deep connections between different data types. The model is trained using a contrastive learning framework to generate feature representations that reflect the intrinsic connections between the modalities. Unlike traditional methods, this technology does not rely on explicit corresponding annotations, reducing the need for large-scale annotated datasets and thus improving the model's generalization and adaptability.

[0004] This invention is applicable to a variety of application scenarios, such as visual question answering, image captioning, cross-modal retrieval, multimodal sentiment analysis, and human-computer interaction. It not only enables efficient and accurate processing and understanding of new media information, but also addresses diverse needs such as intelligence mining, public opinion monitoring, topic tracking, and brand reputation analysis. This innovative technical solution significantly enhances the efficiency and accuracy of multimodal data analysis, providing strong support for intelligent information processing. Summary of the Invention

[0005] (1) Technical issues to be solved

[0006] The technical problem to be solved by the present invention is how to provide a multimodal feature alignment method based on implicit feature space to address the deficiencies in multimodal data processing in the prior art and the limitations of multimodal data in feature representation and alignment.

[0007] (2) Technical solution

[0008] In order to solve the above technical problems, the present invention proposes a multimodal feature alignment method based on implicit feature space, which includes the following steps:

[0009] Step 1: Deploy the DeepSeek R1 large model locally;

[0010] Step 2: Fine-tuning the large model with multimodal data

[0011] Input the original text data into DeepSeek to obtain the text features H of the text information T ;

[0012] The text and audio of the original data are input to DeepSeek in the form of text-audio pairs. According to the text prompt, DeepSeek outputs the audio feature H after the text prompt requirement. A ;

[0013] Input the original data text and pictures into DeepSeek in the form of text-picture pairs, and let DeepSeek output the image features H after the text prompt requirements according to the text prompt. I ;

[0014] Based on the open source large model DeepSeek, fine-tuned using the GRPO algorithm;

[0015] Step 3: Cross-modal Attention Mechanism

[0016] For the above text, image, and speech features, a cross-modal attention layer is introduced to enable different modalities to learn from each other and strengthen the relationship between them;

[0017] Step 4: Feature fusion and mapping

[0018] The features of all modalities are mapped into the shared latent space z through a linear transformation.

[0019] (3) Beneficial effects

[0020] The present invention proposes a multimodal feature alignment method based on implicit feature space. The present invention proposes a multimodal feature alignment method based on implicit feature space, and the beneficial effects are reflected in the following aspects:

[0021] (1) In multimodal information processing, with the powerful feature extraction and semantic understanding capabilities of large models, semantic feature extraction of heterogeneous data such as images and audio can be achieved with text as the center. Specifically, through pre-trained large language models and multimodal interfaces, key features related to text descriptions can be extracted from original images or audio, while ignoring irrelevant or redundant information. At the same time, based on text guidance, semantic alignment and local correction can be performed on visual elements in images or speech content in audio, such as enhancing the recognition of specific objects and correcting ambiguous words in speech recognition. This method not only improves the consistency and accuracy of cross-modal data, but also provides strong support for building text-driven intelligent content editing systems.

[0022] (2) We designed an architecture that combines a cross-modal attention mechanism to capture the complex relationships between different modalities. By introducing self-attention and cross-modal attention layers, we not only enhance the model’s understanding of the internal information of each modality, but also ensure the semantic consistency of data from different modalities in a shared latent space. This enables the model to demonstrate its association discovery capabilities in subsequent cross-modal tasks, such as visual question answering, sentiment analysis, and cross-modal retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is the overall framework of the multimodal feature alignment method based on implicit feature space in the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below with reference to the accompanying drawings and examples.

[0025] In order to solve the above technical problems, the present invention proposes a multimodal feature alignment method based on implicit feature space, which includes the following steps:

[0026] Step 1: Deploy the DeepSeek R1 large model locally

[0027] Use open source frameworks such as DeepSeek for model loading and inference. Leverage specialized hardware (such as GPUs, CPUs, and TPUs) to accelerate computation. Deploy lightweight versions of models to accommodate resource-constrained environments.

[0028] Step 2: Fine-tuning the large model with multimodal data

[0029] Input the original text data into DeepSeek to obtain the text features H of the text information T ;

[0030] The text and audio of the original data are input to DeepSeek in the form of text-audio pairs. According to the text prompt, DeepSeek outputs the audio feature H after the text prompt requirement. A ;

[0031] Input the original data text and pictures into DeepSeek in the form of text-picture pairs, and let DeepSeek output the image features H after the text prompt requirements according to the text prompt. I .

[0032] Based on the open-source DeepSeek large-scale model, the GRPO algorithm was used for fine-tuning. The GRPO algorithm consists of three models: a policy model, a reference model, and a reward model (RM). The policy model is a pre-trained DeepSeek R1 model, which is subsequently fine-tuned through reinforcement learning to enable editing capabilities from text to audio and text to images. The reference model is also a pre-trained DeepSeek R1 model, which is used to constrain the differences between the policy model and the reference model. The reward model is a separately trained model whose purpose is to score the rationality of the images and audio output by the policy model. Fine-tuning uses the reinforcement learning training model of the Group Relative Policy Optimization (GRPO) algorithm, which directly enables the model to autonomously develop mapping capabilities through a pure reinforcement learning process.

[0033] Step 3: Cross-modal Attention Mechanism

[0034] For the above text, image, and speech features, a cross-modal attention layer is introduced to enable mutual learning between different modalities and strengthen the relationship between them. The attention representations between text-audio and text-image are: H TA 、H TI and H AI ;

[0035] Step 6: Feature fusion and mapping

[0036] In order to align the features of different modalities in the shared latent space, the features of all modalities are mapped into the shared latent space z through a linear transformation.

[0037] Example 1:

[0038] This invention aims to address the shortcomings of existing multimodal data processing techniques by providing a multimodal feature alignment method based on implicit feature space. This technology efficiently integrates multiple modal information, including text, images, and audio, to enhance the ability to extract key video information and analyze graphic data. It also optimizes computational models for efficient processing and adapts to cross-domain application requirements, while reducing the need for annotated data and resources. This provides a more accurate, flexible, and efficient solution for subsequent analysis and processing tasks.

[0039] The present invention relates to a multimodal feature alignment method based on implicit feature space, which belongs to the fields of artificial intelligence, multimodal data processing, feature representation and alignment. The present invention enhances key information extraction and data analysis capabilities by efficiently integrating multiple modal information such as text, images, and audio, and adapts to cross-domain application needs. Specifically,

[0040] This solution first deploys a large model locally, using open-source frameworks such as DeepSeek R1 for model loading and inference. It leverages specialized hardware such as GPUs and CPUs to accelerate computation, while also supporting lightweight deployment for resource-constrained environments. It then employs a large-model architecture based on DeepSeek R1, integrating key technologies such as a mixture of experts (MoE), load balancing, and a multi-head latent attention (MLA) mechanism to achieve efficient feature extraction for text-image and text-audio combinations. A multimodal input approach is developed, feeding text-audio and text-image pairs into the model. Based on textual cues, the model modifies the audio or image content to meet requirements, laying the foundation for subsequent cross-modal understanding. Self-attention is then independently used to capture long-range dependencies in text, audio, and image, generating high-dimensional feature representations. Self-attention is then applied within each modality to enhance feature expressiveness. Finally, a cross-modal attention layer is introduced to enable mutual learning between modalities and strengthen their connections. To align features from different modalities in a shared latent space, features from all modalities are mapped to the same latent space via a linear transformation.

[0041] This invention not only significantly improves the efficiency and accuracy of multimodal data analysis, but also provides strong support for intelligent information processing. It is suitable for various application scenarios such as visual question answering, sentiment analysis, cross-modal retrieval, and human-computer interaction.

[0042] In order to solve the above technical problems, the present invention provides a multimodal feature alignment method based on implicit feature space, comprising the following steps:

[0043] Step 1: Local deployment of large models

[0044] Prepare a hardware environment that meets computing requirements (such as GPU / TPU) and configure the operating system and software dependencies. Secondly, install the necessary deep learning frameworks (such as PyTorch and TensorFlow) and model libraries (such as Hugging FaceTransformers or SDKs for specific models). Then, download pre-trained models from open source platforms and perform lightweight processing such as quantization and pruning based on the actual application scenario. Finally, integrate the model into the local service and perform inference calls through the API interface or command line to achieve AI applications with stronger data privacy protection and more efficient response.

[0045] Step 2: Fine-tuning the large model with multimodal data

[0046] 2.1 Text-Image Fine-tuning

[0047] Prepare the original image and the prompt text. The text should clearly express the desired modification or adjustment to the image. Use the model to extract features from both the text and the image. This step is the foundation of multimodal processing, enabling data from different modalities to be represented in a unified latent space. Based on the information provided by the text, adjust the image features in the latent space. Then, use the decoder to convert the modified features back into image format. Output the modified image and verify that the desired effect has been achieved.

[0048] 2.2 Fine-tuning of text-audio

[0049] Select the Deepseek R1 pre-trained multimodal model for text-guided audio processing and load it into the environment through the API. Prepare the original audio file and prompt text. Use the model to extract features from the text and audio separately, then find the correspondence between text and audio features in a shared latent space. Finally, convert the modified features back into audio format.

[0050] Step 3: Cross-modal Attention Mechanism

[0051] The introduction of cross-modal attention layer enables different modalities to learn from each other and strengthen the relationship between them. The attention representations between text-audio, text-image and audio-image are: H TA 、H TI and H AI .

[0052] Step 4: Feature fusion and mapping

[0053] In order to align the features of different modalities in the shared latent space, the features of all modalities (H T 、H A and H I ) is mapped into the shared latent space z through a linear transformation.

[0054] Example 2:

[0055] This paper discloses a multimodal feature alignment method based on implicit feature space, which mainly addresses the limitations of existing technologies in feature representation and alignment of multimodal data (such as text, audio and images), including: (1) cross-modal semantic gap: traditional methods have difficulty in effectively capturing the deep semantic associations between different modal data, resulting in poor results in cross-modal tasks (such as retrieval and generation). (2) Insufficient feature expression: single-modal feature extraction methods often cannot fully express the information in complex scenes, which limits the model's understanding ability and generalization performance.

[0056] This paper proposes a multimodal feature alignment method based on implicit feature space. This framework leverages DeepSeek's powerful multimodal understanding capabilities and combines self-attention and cross-modal attention mechanisms to capture complex relationships within and between modalities. Furthermore, it optimizes model performance by setting a reinforcement learning reward and penalty function. The specific steps are as follows:

[0057] Step 1: Local deployment of large models

[0058] Prepare a hardware environment that meets computing requirements (such as GPU / TPU) and configure the operating system and software dependencies. Secondly, install the necessary deep learning frameworks (such as PyTorch and TensorFlow) and model libraries (such as Hugging FaceTransformers or SDKs for specific models). Then, download pre-trained models from open source platforms and perform lightweight processing such as quantization and pruning based on the actual application scenario. Finally, integrate the model into the local service and perform inference calls through the API interface or command line to achieve AI applications with stronger data privacy protection and more efficient response.

[0059] Step 2: Fine-tuning the large model with multimodal data

[0060] 2.1 Text-Image Fine-tuning

[0061] Prepare the original image and prompt text. The text should clearly express the desired modification or adjustment to the image. Use the model to extract features from both the text and the image. This step is fundamental to multimodal processing, enabling data from different modalities to be represented in a unified latent space. Based on the information provided by the text, the image features are adjusted in the latent space. The decoder then converts the modified features back into an image format. The modified image is output and verified to have achieved the desired effect using the reinforcement learning algorithm GPRO.

[0062] 2.2 Fine-tuning of text-audio

[0063] Select the pre-trained multimodal model for Deepseek R1 text-guided audio processing and load it into the environment through the API. Prepare the original audio file and prompt text. Use the model to extract features from the text and audio separately, then find the correspondence between text features and audio features in a shared latent space. Finally, convert the modified features back into audio format. Use the reinforcement learning algorithm GPRO to verify that the expected results are achieved.

[0064] Step 3: Cross-modal attention mechanism

[0065] After DeepSeek fine-tunes the generated audio and image features, a cross-modal attention layer is introduced to enable different modalities to learn from each other and strengthen the association between them. The attention representations between text-audio, text-image, and audio-image are: H TA 、H TI and H AI , which is calculated as follows:

[0066] H TA =CrossModalAttention(H T ,H A )

[0067] H TI =CrossModalAttention(H T ,H I )

[0068] H AI =CrossModalAttention(H A ,H I )

[0069] Cross-modal attention can be implemented through a query-key-value (QKV) structure, namely:

[0070]

[0071] Among them, Q x , K y , V y are query, key, and value matrices, respectively, d k is the key dimension.

[0072] Specifically, in the cross-modal attention calculation process between text and audio, the query matrix Q of the text modality is T , the key matrix K of the audio mode A , the value matrix V of the audio mode A They are:

[0073]

[0074] in, and They are used to transform the text features H T and audio feature H A The weight matrix mapped to the query, key and value space. Therefore, the cross-modal attention representation H between text and audio TA for:

[0075]

[0076] In the cross-modal attention calculation process between text and image, the query matrix Q of the text modality T , the key matrix K of the image modality I , the value matrix V of the image modality I They are:

[0077]

[0078] in, and They are used to transform the text features H T and image features H I The weight matrix mapped to the query, key and value space. Therefore, the cross-modal attention representation H between text and image TI for:

[0079]

[0080] In the cross-modal attention calculation process between audio and image, the query matrix Q of the audio modality A , the key matrix K of the image modality I , the value matrix V of the image modality I They are:

[0081]

[0082]

[0083] in, and They are used to transform the audio features H A and image features H I The weight matrix mapped to the query, key and value space. Therefore, the cross-modal attention representation H between audio and image AI for:

[0084]

[0085] Step 4: Feature fusion and mapping

[0086] In order to align the features of different modalities in the shared latent space, we map the features of all modalities to the shared latent space z through linear transformation. The feature representations of text, audio and video in the shared latent space are z T 、z A and z I , the calculation process is as follows:

[0087] z T =W T H T +W TI HI +W TA H A

[0088] z A =W A H A +W AT H T +W AI H I

[0089] z I =W I H I +W IT H T +W IA H A

[0090] Where W T , W A , W I is the weight moment of each mode, and W TI 、W TA 、W AT 、W AI 、W IT 、W IA It is used to adjust the contribution ratio of cross-modal features. This not only allows the features of each modality to be aligned in the shared space, but also preserves the information gain brought by the cross-modal attention mechanism.

[0091] The loss function L is composed of the task-specific loss L task and contrast loss L contrastive constitute:

[0092] L=w1L task +w2L contrastive

[0093] Among them, w1 and w2 are weight coefficients used to balance the importance of different losses.

[0094] Task-specific loss refers to the loss function used to complete a specific task, which directly optimizes the performance of the model on that task.

[0095] The contrastive loss for feature alignment ensures that data from different modalities are effectively aligned in a shared latent space, promoting the learning of implicit features and cross-modal consistency. Together, these two form a comprehensive loss function that ensures the model's effectiveness on specific tasks while improving the overall performance of multimodal data processing.

[0096] The contrast loss function encourages the similarity between positive samples and increases the distance between negative samples. The contrast loss L contrastive It can be expressed by the following formula.

[0097]

[0098] Where D represents the distance metric between two vectors (cosine similarity), z x is a sample in the latent space z, is with z x Matched positive samples, is the jth sample in the negative sample set, and N represents the number of samples in the negative sample set. τ is a temperature parameter used to adjust the degree of influence of the similarity score. Positive samples refer to samples that have the same identity or are highly correlated with a given sample. In other words, they come from different perspectives, different modalities, or different time points of the same instance or object. For example, taking text-image pairing as an example, if a sample is a picture, its corresponding positive sample may be the text description of this picture. Negative samples refer to samples that are unrelated to the given sample or come from different identities. This means that they do not belong to the same instance or object. In the above text-image pairing example, any other image description or text that is not related to the current image can be regarded as a negative sample of the image. For positive sample pairs Since they are matched, we expect their distance in the shared latent space to be Smaller. For negative samples Since they are not matched, their distance in the shared latent space is expected to be Larger. The temperature parameter τ controls the degree of scaling of the similarity scores. Smaller values ​​of τ result in more extreme similarity scores, causing the model to focus more on distinguishing between positive and negative samples; larger values ​​of τ smooth the score distribution, helping the model learn on more difficult tasks. The entire formula maximizes the similarity between positive sample pairs while minimizing the similarity between negative sample pairs. In this way, the contrastive loss function effectively encourages similarity between positive samples and distances negative samples, achieving better feature alignment and cross-modal consistency.

[0099] The present invention discloses a multimodal feature alignment method based on implicit feature space, the main advantages of which are reflected in the following aspects:

[0100] (1) In multimodal information processing, the powerful feature extraction and semantic understanding capabilities of large models can be used to achieve semantic feature alignment of heterogeneous data such as images and audio, centered on text. Specifically, through the use of pre-trained large language models and multimodal interfaces, key features related to the text description can be extracted from the original image or audio, while ignoring irrelevant or redundant information.

[0101] (2) We designed an architecture that combines a cross-modal attention mechanism to capture the complex relationships between different modalities. By introducing self-attention and cross-modal attention layers, we not only enhance the model’s understanding of the internal information of each modality, but also ensure the semantic consistency of data from different modalities in a shared latent space. This enables the model to demonstrate its powerful association discovery capabilities in cross-modal tasks such as visual question answering, sentiment analysis, and cross-modal retrieval.

[0102] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A multimodal feature alignment method based on implicit feature space, characterized in that: The method comprises the following steps: Step 1: Deploy the DeepSeek R1 large model locally; Step 2: Fine-tuning the large model with multimodal data Input the original text data into DeepSeek to obtain the text features H of the text information T ; The text and audio of the original data are input to DeepSeek in the form of text-audio pairs. According to the text prompt, DeepSeek outputs the audio feature H after the text prompt requirement. A ; Input the original data text and pictures into DeepSeek in the form of text-picture pairs, and let DeepSeek output the image features H after the text prompt requirements according to the text prompt. I ; Based on the open source large model DeepSeek, fine-tuned using the GRPO algorithm; Step 3: Cross-modal Attention Mechanism For the above text, image, and speech features, a cross-modal attention layer is introduced to enable different modalities to learn from each other and strengthen the relationship between them; Step 4: Feature fusion and mapping The features of all modalities are mapped into the shared latent space z through a linear transformation.

2. The multimodal feature alignment method based on implicit feature space according to claim 1, wherein: In step 2, the DeepSeek R1 large model constructs a multimodal input method, inputs text-audio pairs and text-image pairs into the model, and guides the model to modify the audio or image content that meets the requirements based on the text prompts. Then, the text, audio, and image independently use self-attention to capture long-distance dependencies and generate high-dimensional feature representations, and apply the self-attention mechanism within each modality to enhance feature expressiveness.

3. The multimodal feature alignment method based on implicit feature space according to claim 1, wherein: In the step 2, the GRPO algorithm is used for fine-tuning based on the DeepSeek large model, including: the GRPO algorithm includes three models, namely the strategy model, the reference model and the reward model RM; the strategy model is a pre-trained DeepSeek R1 model, which is fine-tuned through subsequent reinforcement learning to enable it to have the editing capabilities from text to audio and text to images; the reference model is also a pre-trained model DeepSeek R1, which is used to limit the difference between the constraint strategy model and the reference model; the reward model is a separately trained model, and its purpose is to give a score for the rationality of the pictures and audio output by the strategy model; the reinforcement learning training mode of the group relative strategy optimization algorithm GRPO is adopted during fine-tuning, and the model is directly enabled to autonomously develop mapping capabilities through a pure reinforcement learning process.

4. The multimodal feature alignment method based on implicit feature space according to claim 3, wherein: When fine-tuning text-image, prepare the original image and prompt text. The text should clearly express the desired modification or adjustment to the image. Use the model to extract features from the text and image separately. Based on the information guided by the text, the image features are adjusted in the latent space, and then the modified features are converted back into image format through the decoder. The modified image is output and checked to see if the expected effect is achieved.

5. The multimodal feature alignment method based on implicit feature space according to claim 3, wherein: When fine-tuning for text-audio, select the pre-trained multimodal model of DeepseekR1 text-guided audio processing and load it into the environment through the API; prepare the original audio file and prompt text, use the model to extract features from the text and audio respectively, then find the correspondence between text features and audio features in the shared latent space, and finally convert the modified features back into audio format.

6. The multimodal feature alignment method based on implicit feature space according to any one of claims 1 to 5, characterized in that: After DeepSeek fine-tunes the generated audio and image features, a cross-modal attention layer is introduced to enable different modalities to learn from each other and strengthen the association between them; the attention representations between text-audio, text-image, and audio-image are: H TA 、H TI and H AI , which is calculated as follows: H TA =CrossModalAttention(H T ,H A ) H TI =CrossModalAttention(H T ,H I ) H AI =CrossModalAttention(H A ,H I ) Cross-modal attention is implemented through the query-key-value QKV structure, namely: Among them, Q x , K y , V y are query, key, and value matrices, respectively, d k is the key dimension; Specifically, in the cross-modal attention calculation process between text and audio, the query matrix Q of the text modality is T , the key matrix K of the audio mode A , the value matrix V of the audio mode A They are: in, and They are used to transform the text features H T and audio feature H A The weight matrix mapped to the query, key, and value space; thus, the cross-modal attention representation H between text and audio TA for: In the cross-modal attention calculation process between text and image, the query matrix Q of the text modality T , the key matrix K of the image modality I , the value matrix V of the image modality I They are: in, and They are used to transform the text features H T and image features H I The weight matrix mapped to the query, key and value space; therefore, the cross-modal attention representation H between text and image TI for: In the cross-modal attention calculation process between audio and image, the query matrix Q of the audio modality A , the key matrix K of the image modality I , the value matrix V of the image modality I They are: in, and They are used to transform the audio features H A and image features H I The weight matrix mapped to the query, key and value space; thus, the cross-modal attention representation H between audio and image AI for:

7. The multimodal feature alignment method based on implicit feature space according to claim 6, characterized in that: In step 4, the features of all modalities are mapped to the shared latent space z through linear transformation; the feature representations of text, audio and video in the shared latent space are z T 、z A and z I , the calculation process is as follows: z T =W T H T +W TI H I +W TA H A z A =W A H A +W AT H T +W AI H I z I =W I H I +W IT H T +W IA H A Where W T , W A , W I is the weight moment of each mode, and W TI 、W TA 、W AT 、W AI 、W IT 、W IA It is used to adjust the contribution ratio of cross-modal features.

8. The multimodal feature alignment method based on implicit feature space according to claim 7, wherein: The loss function L of this method is composed of the task-specific loss L task and contrast loss L contrastive constitute: L=w1L task +w2L contrastive Among them, w1 and w2 are weight coefficients used to balance the importance of different losses; Task-specific loss refers to the loss function used when completing a specific task, which is used to directly optimize the performance of the model on the task; the contrastive loss of feature alignment is used to ensure that data of different modalities are effectively aligned in the shared latent space, promoting the learning of implicit features and cross-modal consistency.

9. The multimodal feature alignment method based on implicit feature space according to claim 8, characterized in that: The contrast loss function encourages the similarity between positive samples and increases the distance between negative samples. The contrast loss L contrastive Expressed using the following formula: Where D represents the distance metric between two vectors, z x is a sample in the latent space z, is with z x Matched positive samples, is the jth sample in the negative sample set, N represents the number of samples in the negative sample set; τ is the temperature parameter, which is used to adjust the influence of the similarity score; positive samples refer to samples with the same identity or high correlation with the given sample, or from different perspectives, different modalities or different time points of the same instance or object, and negative samples refer to samples that are unrelated to the given sample or come from different identities, which means they do not belong to the same instance or object.

10. The multimodal feature alignment method based on implicit feature space according to claim 9, characterized in that: The distance metric is calculated using cosine similarity.

Images