Multimodal embedding representation learning method based on semantic enhancement in data space

Abstract

The invention provides a multi-modal embedded representation learning method based on semantic enhancement, and belongs to the field of multi-modal data processing.The method comprises the steps that firstly, multi-modal text, image, video and audio data are obtained, data preprocessing is carried out, and format conversion and normalization processing are completed; then extracting features in the processed text, image and audio data, and generating an initial modal embedding representation; a semantic enhancement strategy is introduced into image, video and audio data, semantic consistency among multiple modes is enhanced, and unified multi-mode vector representation is generated; the embedded representation is normalized, and semantic consistency of all modals in a unified vector space is ensured; by introducing video key frame extraction, image optical character recognition and audio-to-text technologies, semantic supplementation among multi-modal data is further enhanced, and distribution of embedded representation is optimized, so that the embedded representation learning effect of multi-modal information is improved.

Description

Technical Field

[0001] The present invention belongs to the field of multimodal data processing, and specifically, relates to a method for multimodal embedding representation learning based on semantic enhancement in a data space. Background Art

[0002] With the development of artificial intelligence technology, especially the breakthroughs in fields such as computer vision, speech recognition, and natural language processing, multimodal learning has gradually become an important research direction. Multimodal learning aims to improve the ability to understand complex data and decision-making ability by integrating data from different modalities such as text, images, videos, and audio. In the process of multimodal learning, a core issue is how to convert data from different modalities into a unified representation for subsequent analysis and processing.

[0003] In recent years, the data space, as an emerging concept of data management and sharing, has attracted wide attention. It provides a standardized way of data storage, organization, and access, enabling data from different sources and in different formats to be efficiently integrated and utilized. In the data space, multimodal data can be stored in a structured or unstructured form and annotated with a unified semantic description, thus supporting more efficient multimodal data fusion and representation learning. Existing traditional multimodal learning methods usually rely on separate representations of different modalities and are fused through certain forms of alignment techniques. However, these methods still face some challenges.

[0004] One is the semantic inconsistency between modalities. Data from different modalities (such as text, images, audio, videos, etc.) usually adopt different representation methods and are in different vector spaces. Therefore, how to effectively map the data of these modalities into a unified vector space and ensure the semantic consistency between them has become a major problem in multimodal learning.

[0005] Another is the lack and incompleteness of information. In practical applications, data from different modalities are often incomplete or limited by the acquisition environment. For example, in a video, some text information may not be effectively extracted, or due to the influence of audio signal noise, the speech information in the audio is difficult to identify. In this case, how to make up for these missing information through external semantic enhancement information is the key to improving the effect of multimodal learning.

[0006] The third is insufficient semantic enhancement. Although existing multimodal learning methods can effectively align the features of different modalities, few methods can fully introduce external semantic enhancement information. Traditional methods usually only rely on the existing modal data and ignore the potential semantic complementarity between modalities. For example, in image data, text information is usually difficult to directly extract, and if the speech content in audio data cannot be correctly recognized, it may lead to the loss of the entire modal information.

[0007] To address these problems, existing technologies typically solve the multi-modal representation problem through simple modal alignment or joint training. However, their ability to handle semantic consistency and information supplementation between modalities remains limited, often unable to ensure complete semantic alignment between modalities. Summary of the Invention

[0008] To address the problems of the existing technology, the present invention proposes a multi-modal embedding representation learning method based on semantic enhancement in the data space. By introducing video key frame extraction, image optical character recognition, and audio speech-to-text technology, the semantic supplementation between multi-modalities is enhanced, thereby optimizing the distribution of the embedding representation. The introduction of the data space enables multi-modal data from different sources to be efficiently fused in a unified environment, further improving the effect of multi-modal representation learning.

[0009] Multi-modal embedding representation learning method based on semantic enhancement in the data space: The method specifically includes the following steps:

[0010] Step 1, multi-modal data preprocessing, performing format conversion and normalization processing on the input multi-modal data to ensure the consistency of different modal data;

[0011] Step 2, modal feature extraction, extracting the features of the preprocessed multi-modal data respectively to generate preliminary modal embedding representations;

[0012] Step 3, semantic enhancement strategy, performing semantic enhancement on the preliminary feature embeddings. By introducing semantic supplementary information to some modal data, the semantic consistency of multi-modal data is enhanced to generate a unified multi-modal vector representation;

[0013] Step 4, unified vector space alignment, by constructing a projection module and combining the results of semantic enhancement, aligning all modal embeddings to a unified vector space to ensure the semantic consistency of each modality within the unified vector space and generating the final multi-modal representation.

[0014] Furthermore, in Step 1, the multi-modal data includes text, images, videos, and audio;

[0015] Step 1.1, text data processing, performing standardization processing on the text data, unifying the encoding format, and removing irrelevant characters or noise data;

[0016] Step 1.2, image data processing, performing color channel standardization processing on the image data;

[0017] Step 1.3, video data processing, splitting the video data into independent frames, extracting key frames according to time intervals or scene change detection, and performing color channel standardization processing;

[0018] Step 1.4, Audio data processing, converting the audio data into a time-domain or frequency-domain representation form.

[0019] Furthermore, in Step 2, the CLIP model and the VGGish model need to be trained first before extracting features;

[0020] The pre-training process of the CLIP model is as follows:

[0021] First, use pre-aligned image-text pairs to fine-tune the CLIP model.

[0022] Use the original BERT and ViT models to obtain the normalized embeddings of the text and the normalized embeddings of the images in N pairs of pre-aligned image-text pairs respectively; calculate the similarity between each text and each image;

[0023] Use the contrastive learning loss to gradually iteratively train the original BERT and ViT models to improve the similarity between the aligned text and image; reduce the similarity between the unaligned text and image;

[0024] Merge the trained BERT and ViT models as the final CLIP model.

[0025] Furthermore, the pre-training process of the VGGish model is as follows:

[0026] Use pre-aligned audio-text pairs to fine-tune the VGGish model:

[0027] Use the BERT model in the obtained final CLIP model and the original VGGish model to obtain the normalized embeddings of the text and the normalized original embeddings of the audio in several pairs of pre-aligned audio-text pairs respectively, and map the embedding of each audio through a fully connected layer and the activation function ReLU; calculate the similarity between each text and each audio;

[0028] Use the contrastive learning loss to gradually iteratively train the VGGish, fully connected layer and activation function to improve the similarity between the aligned text and audio, and reduce the similarity between the unaligned text and audio;

[0029] Take the trained VGGish, fully connected layer and activation function as the final model for embedding audio files.

[0030] Furthermore, in Step 2,

[0031] Step 2.1, Text feature extraction, use the BERT module in the pre-trained CLIP model to extract the text data features processed in Step 1.1 and generate text embedding representations;

[0032] Step 2.2, Image Feature Extraction: Use the ViT module in the pre-trained CLIP model to extract the image features processed in Step 1.2 and generate an image embedding representation:

[0033] Step 2.3, Video Feature Extraction: Use the ViT module in the pre-trained CLIP model to extract the key frame features processed in Step 1.3 and generate a video embedding representation;

[0034] Step 2.4, Audio Feature Extraction: Use the pre-trained VGGish model to extract the audio features processed in Step 1.4 and generate a temporary audio embedding representation.

[0035] Furthermore, in Step 3, it includes:

[0036] Step 3.1, Text Extraction from Video Key Frames: Perform OCR on the text regions in the video key frames to extract text information as semantic supplement. If no text is extracted, skip semantic enhancement;

[0037] Step 3.2: Text Extraction from Images: Perform OCR on the text regions in the images to extract text information as semantic supplement. If no text is extracted, skip semantic enhancement:

[0038] Step 3.3, Text Conversion of Audio: Use speech recognition technology to convert the audio data into text embeddings. If there is text content, enter semantic enhancement. If no text content is recognized, skip semantic enhancement;

[0039] Step 3.4, Input the text extracted in Steps 3.1 to 3.3 for semantic enhancement into the trained CLIP model to generate feature embeddings in a unified vector space as semantic enhancement representations.

[0040] Furthermore, in Step 4, it includes:

[0041] Step 4.1, Construct a projection module containing structures such as fully connected layers and activation functions to learn the alignment relationships between modalities,

[0042] Step 4.2: Through the aligned text-audio pairs, train the projection module constructed in Step 4.1 separately, using the cosine similarity loss to maximize the similarity between the audio embedding and the text embedding. If there is noise in the embedding content, improve the stability of the embedding through a noise reduction algorithm;

[0043] Step 4.3: Use the projection module trained in Step 4.2 to project the temporary audio embedding representation generated in Step 2.4 into a unified vector space as the final audio embedding representation. If there is noise in the embedding content, improve the stability of the embedding through a noise reduction algorithm;

[0044] Step 4.4: Align the modal embeddings, optimize and train the embeddings using contrastive loss, and perform final representation learning on the input data through the trained joint embedding model to generate aligned multi-modal embedding representations, ensuring semantic consistency between modalities.

[0045] A multi-modal embedding representation learning system based on semantic enhancement in the data space

[0046] The system includes a preprocessing module, a feature extraction module, a semantic enhancement module, and a multi-modal representation module;

[0047] The preprocessing module performs format conversion and normalization processing on the input multi-modal data to ensure the consistency of different modal data;

[0048] The feature extraction module extracts the features of the preprocessed multi-modal data respectively to generate preliminary modal embedding representations;

[0049] The semantic enhancement module performs semantic enhancement on the preliminary feature embeddings. By introducing semantic supplementary information to some modal data, it enhances the semantic consistency of the multi-modal data and generates a unified multi-modal vector representation;

[0050] The multi-modal representation module unifies and aligns the vector space, constructs a projection module, and combines the results of semantic enhancement to align all modal embeddings into a unified vector space, ensuring the semantic consistency of each modality within the unified vector space and generating the final multi-modal representation.

[0051] An electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the above method.

[0052] A computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, they implement the steps of the above method.

[0053] Advantages of the present invention

[0054] Compared with the prior art, the present invention makes up for the deficiencies in semantic consistency and information supplementation in traditional multi-modal embedding representation learning methods. By using the semantic enhancement strategy, it not only optimizes the alignment effect of semantics between modalities but also realizes the efficient fusion of multi-modal embedding representations in a unified vector space, thus significantly improving the expression ability of multi-modal data.

[0055] The present invention further enhances the semantic supplementation between multi-modal data and optimizes the distribution of embedding representations by introducing video key frame extraction, image optical character recognition, and audio speech-to-text technologies, thereby improving the learning effect of multi-modal information embedding representation. Brief description of the drawings

[0056] Figure 1 This is a flowchart of the semantic-enhanced multi-modal embedding representation learning method in the data space of the present invention, showing the entire process from model training to the generation of multi-modal semantic-enhanced embedding representations;

[0057] Figure 2 This is a schematic diagram of the model structure of the semantic-enhanced multi-modal embedding representation learning method in the data space;

[0058] Figure 3 This is a schematic flowchart of the model training process;

[0059] Figure 4 This is a schematic flowchart of the generation process of the basic modal representations in the unified vector space;

[0060] Figure 5 This is a schematic flowchart of the generation process of the semantic-enhanced modal representations in the unified vector space. Detailed implementation manners

[0061] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0062] The experimental methods used in the following embodiments are all conventional methods unless otherwise specified. The materials, reagents, methods, and instruments used are all conventional materials, reagents, methods, and instruments in the art unless otherwise specified, and those skilled in the art can obtain them through commercial channels.

[0063] The present invention proposes a semantic-enhanced multi-modal embedding representation learning method in the data space:

[0064] Step 1, multi-modal data preprocessing, performing format conversion and normalization processing on the input multi-modal data to ensure the consistency of different modal data;

[0065] The multi-modal data includes text, images, videos, and audio;

[0066] Step 1.1: Text data processing, standardizing the text data, unifying the encoding format, and removing irrelevant characters or noise data, specifically including the following steps:

[0067] Step 1.1.1: Remove special symbols and retain semantic-related content;

[0068] Step 1.1.2: Encode all texts into a unified character set.

[0069] Step 1.2: Image data processing. Normalize the color channels of the image data to make the image features comparable. The specific steps are as follows:

[0070] Step 1.2.1: Normalize the RGB channels by subtracting the channel mean and dividing by the standard deviation;

[0071] Step 1.2.2: Resize the image to fit the input of the subsequent model.

[0072] Step 1.3: Video data processing. Split the video data into frames and select some key frames for processing. The specific steps are as follows:

[0073] Step 1.3.1: Extract key frames according to time interval or scene change detection;

[0074] Step 1.3.2: Perform color channel normalization on the key frames to be consistent with the image data.

[0075] Step 1.4: Audio data processing. Convert the audio data into time-domain or frequency-domain representation forms to provide input for subsequent feature extraction. The specific steps are as follows:

[0076] Step 1.4.1: Resample the audio file to 16kHz mono audio to ensure a unified sampling rate and number of channels;

[0077] Step 1.4.2: Apply the short-time Fourier transform to the audio signal to generate a spectrogram representation of the audio;

[0078] Step 1.4.3: Map the spectrogram to a 64-order Mel filter bank, calculate the Mel spectrogram, and extract the frequency features perceptible to the human ear.

[0079] Step 2, Modal feature extraction. Extract the features of the multi-modal data preprocessed in Step 1 respectively to generate a preliminary modal embedding representation;

[0080] Before extracting features, the CLIP model and the VGGish model need to be trained first.

[0081] First, use the pre-aligned image-text pairs to fine-tune (pre-train) the CLIP model. Specifically:

[0082] Use the original BERT and ViT models to obtain the normalized embeddings [T1, T2, …, T N and the normalized embeddings [I1, I2, …, I N of the text and images in N pairs of pre-aligned image-text pairs respectively, where i = 1, 2, …, N;

[0083] Calculate the similarity T between each text and each image i ·I j , where i, j = 1, 2, …, N;

[0084] Use the contrastive learning loss to iteratively train the original BERT and ViT models step by step to improve the similarity T between the aligned text and image i ·I i ; Reduce the similarity T between the misaligned text and image i ·I j (i ≠ j);

[0085] Merge the trained BERT and ViT models as the final CLIP model.

[0086] Fine-tune (pre-train) the VGGish model using pre-aligned audio-text pairs, specifically:

[0087] Use the BERT model in the obtained final CLIP model and the original VGGish model respectively to obtain the normalized embeddings [T1 ′ , T2 ′ , …, T M ′ of the text and the normalized original embeddings [A1 ′ , A2 ′ , …, A M ′ of the audio in M pre-aligned audio-text pairs, where

[0088] Map the embedding A of each audio i ′ through a fully connected layer and the activation function ReLU to

[0089] Calculate the similarity T between each text and each audio i ′ ·A j , where i, j = 1, 2, …, M;

[0090] Use the contrastive learning loss to iteratively train VGGish, the fully connected layer and the activation function to improve the similarity T between the aligned text and audio i ′ ·A i , and reduce the similarity T between the misaligned text and audio i ′ ·A j (i ≠ j);

[0091] Use the trained VGGish, fully connected layer and activation function as the final model for embedding audio files.

[0092] Step 2.1: Text feature extraction. Use the BERT module in the pre-trained CLIP model to extract the features of text data, which specifically includes the following steps:

[0093] Step 2.1.1: Tokenize the text input and generate the context semantic embedding representation;

[0094] Step 2.1.2: Encode through the BERT model to obtain the semantic vector, which is used as the initial embedding of the text modality.

[0095] Step 2.2: Image feature extraction. Use the ViT module in the CLIP model to extract the image features, which specifically includes the following steps:

[0096] Step 2.2.1: Divide the image input into blocks (such as 16x16 blocks);

[0097] Step 2.2.2: Encode through the ViT module to generate the global semantic embedding representation of the image.

[0098] Step 2.3: Video feature extraction. Use the ViT module in the pre-trained CLIP model to extract the features of the key frames processed in Step 1.3 to generate the video embedding representation.

[0099] Step 2.3.1: Use the ViT module in the pre-trained CLIP model to extract the features of multiple key frames processed in Step 1.3 respectively to generate the corresponding key frame embedding representations;

[0100] Step 2.3.2: Take the average of the multiple key frame embedding representations obtained in Step 2.3.1 to obtain the video embedding representation;

[0101] Step 2.4: Audio feature extraction. Use the pre-trained VGGish model to extract the audio features, which specifically includes the following steps:

[0102] Step 2.4.1: Input the Mel spectrogram into the VGGish model;

[0103] Step 2.4.2: Extract the audio features through the VGGish model. Each frame is mapped to a 128-dimensional feature vector, and the mean of all frames is used as the initial (temporary) audio embedding representation.

[0104] Step 3: Semantic enhancement strategy. By introducing semantic supplementary information to some modalities, enhance the semantic consistency of the multi-modal data, which specifically includes the following steps:

[0105] Step 3.1: Text extraction from video key frames. Perform OCR on the text regions in the video key frames to extract the text information as semantic supplement, which specifically includes the following steps:

[0106] Step 3.1.1: Use OCR technology to detect and recognize the text information in the key frames of the video. If no text is extracted, skip semantic enhancement;

[0107] Step 3.1.2: Encode the extracted text information into semantic embeddings through the BERT model to generate semantic supplementary embedding representations.

[0108] Step 3.2: Extract text in the image. Perform OCR on the text area in the image to extract text information as semantic supplement. The specific steps are as follows:

[0109] Step 3.2.1: Use OCR technology to detect and recognize the text information in the image. If no text is extracted, skip semantic enhancement;

[0110] Step 3.2.2: Encode the extracted text information into semantic embeddings through the BERT model to generate semantic supplementary embedding representations.

[0111] Step 3.3: Convert text in the audio. Use speech recognition technology to convert audio data into text embeddings. The specific steps are as follows:

[0112] Step 3.3.1: Use speech recognition technology to transcribe the audio into text information. If no text is extracted, skip semantic enhancement;

[0113] Step 3.3.2: Encode the transcribed text information into semantic embeddings through the BERT model to generate semantic supplementary embedding representations.

[0114] Step 3.4: Input the text extracted in Steps 3.1, 3.2, and 3.3 for semantic enhancement into the BERT module of the trained CLIP model to generate feature embeddings in a unified vector space as semantic supplements for the corresponding modalities, and generate semantic enhancement embedding representations.

[0115] Step 3.4.1: Extract the features of the text information obtained in Step 3.2.1 through the BERT module in the pre-trained CLIP model to generate text embedding representations as semantic supplements for the image modality, and generate semantic enhancement embedding representations;

[0116] Step 3.4.2: Extract the features of the text information obtained in Step 3.3 through the BERT module in the pre-trained CLIP model to generate text embedding representations as semantic supplements for the audio modality, and generate semantic enhancement embedding representations;

[0117] Step 3.4.3: Extract the features of the multiple text information obtained in Step 3.2.2 through the BERT module in the pre-trained CLIP model to generate text embedding representations;

[0118] Step 3.4.4: Take the average of the multiple text embedding representations obtained in Step 3.4.3 as the semantic supplement of the video modality to generate a semantically enhanced embedding representation.

[0119] Step 4: Unified vector space alignment. By constructing a projection module, align all modality embeddings to a unified vector space, which specifically includes the following steps:

[0120] Step 4.1: Construct a projection module containing structures such as fully connected layers and activation functions to learn the alignment relationship between modalities, which specifically includes the following steps:

[0121] Step 4.1.1: Add a linear transformation layer to the input embedding representation and map it to a unified dimension;

[0122] Step 4.1.2: Enhance the non-linear mapping ability through an activation function.

[0123] Step 4.2: Individually train the projection module constructed in Step 4.1 with the aligned text-audio pairs. Using the cosine similarity loss, maximize the similarity between the audio embedding and the text embedding. If there is noise in the embedding content, improve the stability of the embedding through a denoising algorithm;

[0124] Step 4.3: Use the projection module trained in Step 4.2 to project the temporary audio embedding representation generated in Step 2.4 into a unified vector space as the final audio embedding representation. If there is noise in the embedding content, improve the stability of the embedding through a denoising algorithm.

[0125] Step 4.4: Align the modality embeddings, optimize and train the embeddings using the contrastive loss, and perform final representation learning on the input data through the trained joint embedding model to generate aligned multi-modal embedding representations, ensuring semantic consistency between modalities, which can be directly used for multi-modal tasks such as recommendation, retrieval, and classification; specifically includes the following steps:

[0126] Step 4.4.1: Construct positive and negative sample pairs and calculate the objective of maximizing the similarity of positive samples and minimizing the similarity of negative samples;

[0127] Step 4.4.2: Optimize the parameters of the projection module so that the semantic representations of different modalities are aligned to a unified vector space.

[0128] Step 4.5: Output the generated embedding representation and store it for subsequent application scenarios to call.

[0129] A multi-modal embedding representation learning system based on semantic enhancement in the data space,

[0130] The system includes a preprocessing module, a feature extraction module, a semantic enhancement module, and a multi-modal representation module;

[0131] The preprocessing module performs format conversion and normalization on the input multimodal data to ensure the consistency of different modal data;

[0132] The feature extraction module extracts the features of the preprocessed multimodal data respectively to generate a preliminary modal embedding representation;

[0133] The semantic enhancement module performs semantic enhancement on the preliminary feature embedding. By introducing semantic supplementary information to some modal data, it enhances the semantic consistency of the multimodal data and generates a unified multimodal vector representation;

[0134] The multimodal representation module unifies and aligns the vector space, constructs a projection module, and combines the results of semantic enhancement to align all modal embeddings to a unified vector space, ensuring the semantic consistency of each modality within the unified vector space and generating the final multimodal representation.

[0135] An electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps of the above method are implemented.

[0136] A computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the steps of the above method are implemented.

[0137] The memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory of the method described in the present invention is intended to include but not limited to these and any other suitable types of memory.

[0138] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired means such as coaxial cables, optical fibers, digital subscriber line (DSL), or wireless means such as infrared, wireless, microwave, etc. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains one or more integrated available media. The available medium can be a magnetic medium such as a floppy disk, hard disk, magnetic tape, an optical medium such as a high-definition digital video disc (DVD), or a semiconductor medium such as a solid state disc (SSD), etc.

[0139] In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in the processor or the instructions in the form of software. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being executed by the hardware processor, or executed by a combination of the hardware and software modules in the processor. The software module can be located in a mature storage medium in the art such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers, etc. This storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps of the above method. To avoid repetition, it will not be described in detail here.

[0140] It should be noted that the processor in the embodiments of the present application can be an integrated circuit chip with signal processing capabilities. In the implementation process, the steps of the above method embodiments can be completed by the integrated logic circuit in the hardware of the processor or instructions in the form of software. The above-mentioned processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps of the above method.

[0141] The above has introduced in detail the method for semantic-enhanced multi-modal embedding representation learning in the data space proposed by the present invention, and elaborated on the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A method for learning multi-modal embedding representation based on semantic enhancement in a data space, characterized in that: The method specifically includes the following steps: Step 1, multi-modal data preprocessing, performing format conversion and normalization processing on the input multi-modal data to ensure the consistency of different modal data; Step 2, modal feature extraction, extracting features from the preprocessed multi-modal data respectively to generate a preliminary modal embedding representation; Step 3, semantic enhancement strategy, performing semantic enhancement on the preliminary feature embedding, introducing semantic supplementary information to some modal data to enhance the semantic consistency of the multi-modal data, and generating a unified multi-modal vector representation; Step 4, unified vector space alignment, by constructing a projection module, combining the results of semantic enhancement, aligning all modal embeddings to a unified vector space, ensuring the semantic consistency of each modality in the unified vector space, and generating a final multi-modal representation.

2. The method according to claim 1, characterized in that: In step 1, the multi-modal data includes text, images, videos and audio; Step 1.1, text data processing, performing standardization processing on the text data, unifying the encoding format, and removing irrelevant characters or noise data; Step 1.2, image data processing, performing color channel standardization processing on the image data; Step 1.3, video data processing, splitting the video data into independent frames, extracting key frames according to time intervals or scene change detection, and performing color channel standardization processing; Step 1.4, audio data processing, converting the audio data into a time domain or frequency domain representation form.

3. The method according to claim 2, wherein: In step 2, the CLIP model and the VGGish model need to be pre-trained before feature extraction; The pre-training process of the CLIP model is as follows: First, use pre-aligned image-text pairs to fine-tune the CLIP model, Use the original BERT and ViT models respectively to obtain the normalized embeddings of the text and the normalized embeddings of the images in N pairs of pre-aligned image-text pairs; Calculate the similarity between each text and each picture; Use the contrastive learning loss to gradually iteratively train the original BERT and ViT models, improve the similarity between the aligned text and images; reduce the similarity between the unaligned text and images; Merge the trained BERT and ViT models as the final CLIP model.

4. The method according to claim 3, wherein: The pre-training process of the VGGish model is as follows: Use pre-aligned audio-text pairs to fine-tune the VGGish model: Use the BERT model in the obtained final CLIP model and the original VGGish model respectively to obtain the normalized embeddings of the text and the normalized original embeddings of the audio in several pairs of pre-aligned audio-text pairs, map the embedding of each audio through a fully connected layer and the activation function ReLU; calculate the similarity between each text and each audio; Use the contrastive learning loss to gradually iteratively train VGGish, the fully connected layer and the activation function, improve the similarity between the aligned text and audio, and reduce the similarity between the unaligned text and audio; Use the trained VGGish, the fully connected layer and the activation function as the final model for embedding audio files.

5. The method according to claim 4, wherein: In step 2, Step 2.1, Text feature extraction: Use the BERT module in the pre-trained CLIP model to extract the features of the text data processed in Step 1.1 and generate a text embedding representation. Step 2.2, Image feature extraction: Use the ViT module in the pre-trained CLIP model to extract the image features processed in Step 1.2 and generate an image embedding representation. Step 2.3, Video feature extraction: Use the ViT module in the pre-trained CLIP model to extract the key frame features processed in Step 1.3 and generate a video embedding representation. Step 2.4, Audio feature extraction: Use the pre-trained VGGish model to extract the audio features processed in Step 1.4 and generate a temporary audio embedding representation.

6. The method according to claim 5, wherein: In Step 3, it includes: Step 3.1, Text extraction from video key frames: Perform OCR on the text regions in the video key frames to extract text information as semantic supplement. If no text is extracted, skip semantic enhancement. Step 3.2: Text extraction from images: Perform OCR on the text regions in the images to extract text information as semantic supplement. If no text is extracted, skip semantic enhancement. Step 3.3, Text conversion of audio: Use speech recognition technology to convert the audio data into text embeddings. If there is text content, enter semantic enhancement. If no text content is recognized, skip semantic enhancement. Step 3.4, Input the text for semantic enhancement extracted in Steps 3.1 to 3.3 into the trained CLIP model to generate feature embeddings in a unified vector space as the semantic enhancement representation.

7. The method according to claim 6, wherein: In Step 4, it includes: Step 4.1, Construct a projection module including a fully connected layer and an activation function, etc., to learn the alignment relationship between modalities. Step 4.2: Through the aligned text-audio pairs, train the projection module constructed in Step 4.1 separately. Use the cosine similarity loss to maximize the similarity between the audio embedding and the text embedding. If there is noise in the embedding content, improve the stability of the embedding through a denoising algorithm. Step 4.3: Use the projection module trained in Step 4.2 to project the temporary audio embedding representation generated in Step 2.4 into a unified vector space as the final audio embedding representation. If there is noise in the embedding content, improve the stability of the embedding through a denoising algorithm. Step 4.4: Align the modality embeddings, optimize and train the embeddings using the contrast loss, and perform final representation learning on the input data through the trained joint embedding model to generate aligned multi-modal embedding representations to ensure semantic consistency between modalities.

8. A system for performing the method of learning multi-modal embedding representation based on semantic enhancement in the data space according to any one of claims 1 to 7, characterized in that: The system includes a preprocessing module, a feature extraction module, a semantic enhancement module, and a multi-modal representation module; The preprocessing module performs format conversion and normalization processing on the input multi-modal data to ensure the consistency of different modality data; The feature extraction module extracts the features of the preprocessed multi-modal data respectively, Generate preliminary modality embedding representations. The semantic enhancement module performs semantic enhancement on the preliminary feature embeddings. By introducing semantic supplementary information to partial modal data, it enhances the semantic consistency of the multi-modal data and generates a unified multi-modal vector representation. The multi-modal representation module uniformly aligns the vector space, constructs a projection module, and combines the results of semantic enhancement to align all modal embeddings to a unified vector space, ensuring the semantic consistency of each modality within the unified vector space and generating the final multi-modal representation.

9. An electronic device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 7.

10. A computer-readable storage medium for storing computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the method described in any one of claims 1 to 7.

Images