A multi-modal data alignment and enhancement method based on semantic consistency

Abstract

This invention belongs to the field of multimodal data processing technology and discloses a multimodal data alignment and enhancement method based on semantic consistency, aiming to solve the problems of semantic gap and insufficient generalization of multimodal data models. The method includes: acquiring original multimodal data on the same semantic topic, performing noise and anomaly processing and format normalization on the multimodal dataset; extracting semantic features of each modality from the multimodal dataset and normalizing and adapting them to construct a shared semantic space; achieving semantic consistency alignment between modal feature vectors of each modality through a cross-modal alignment mechanism; accordingly, performing intra-modal and cross-modal enhancement on each modality and its shared feature vectors, and integrating and deduplicating to obtain an aligned and enhanced multimodal dataset. This invention achieves accurate semantic alignment of multimodal data, improves data diversity, effectively enhances the generalization ability of multimodal learning models, and is applicable to artificial intelligence scenarios that rely on multimodal data collaboration, demonstrating strong practicality.

Description

Technical Field

[0001] This invention relates to the field of multimodal data processing technology, and more specifically to a method for multimodal data alignment and enhancement based on semantic consistency. Background Technology

[0002] As artificial intelligence penetrates complex scenarios, single-modal data can no longer meet the demands, and multimodal data, including video, images, text, and audio, has become the mainstream input format. However, the inherent "modal heterogeneity" of multimodal data leads to a core bottleneck in collaborative applications. Images and videos emphasize spatial and spatiotemporal features through pixel matrices, text emphasizes semantic logic through discrete symbols, and audio emphasizes frequency and temporal features through spectral features. This difference in format creates a "semantic gap." Existing alignment methods have significant limitations. Most only design strategies for pairwise modalities, failing to cover all four modalities and requiring multiple conversions that easily lead to semantic loss. At the same time, existing methods mostly rely on surface feature matching, making it difficult to handle semantic ambiguity. In addition, the raw data generally contains noise and inconsistent formats, and existing preprocessing only targets single-modal denoising or has inconsistent format standards, further exacerbating the semantic gap problem.

[0003] The performance of multimodal learning models is highly dependent on data diversity, but existing data augmentation techniques have significant shortcomings. Current solutions are mostly limited to single-modal augmentation, making it difficult to expand the dimensionality of multimodal data coverage. Furthermore, some cross-modal augmentation methods are prone to semantic shifts, such as generating an image of a brown cat when inputting text about a black dog, or generating audio that doesn't match the original video's actions, rendering the augmented data unusable for model training. In addition, the lack of a unified semantic selection standard for the augmented data makes it prone to duplicate samples.

[0004] In summary, the core of these problems lies in the fact that existing technologies have not constructed a unified multimodal data processing framework from the perspective of semantic consistency. They lack both a precise alignment mechanism that can bridge the semantic gap and an enhancement strategy that can guarantee semantic invariance. Therefore, the current field of multimodal data processing urgently needs a data enhancement method that unifies preprocessing standards, achieves precise multimodal semantic alignment, and guarantees semantic consistency, in order to solve the core problems of semantic gap and insufficient model generalization, and promote the application of multimodal technology in various industries. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal data alignment and enhancement method based on semantic consistency to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A multimodal data alignment and enhancement method based on semantic consistency includes the following steps:

[0008] S1. Obtain raw multimodal data containing video data, image data, text data, and audio data on the same semantic topic, perform noise anomaly processing and format standardization on the raw multimodal data to obtain a multimodal dataset;

[0009] S2. Extract the semantic features of each modality data from the multimodal dataset using a modal encoder, and normalize and adapt the semantic features of each modality data to obtain a modal feature vector set;

[0010] S3. Construct a shared semantic space, map the modal feature vectors of each modal data in the modal feature vector set to the shared semantic space, and achieve semantic consistency alignment between the modal feature vectors of each modal data corresponding to the same semantic content through a cross-modal alignment mechanism to obtain a multimodal shared feature set;

[0011] S4. Perform intra-modal augmentation on the modal data in the multimodal dataset and cross-modal augmentation on the shared feature vectors of the modal data in the multimodal shared feature set to obtain augmented data including intra-modal augmented data and cross-modal augmented data. Based on this, integrate and deduplicate the augmented data to obtain the aligned augmented multimodal dataset.

[0012] Preferably, the method for noise anomaly processing of the original multimodal data is as follows:

[0013] Image data noise anomaly processing: A noise detection algorithm based on gray-level histogram statistics and pixel neighborhood difference analysis is used to identify noise types in the image data, including salt-and-pepper noise, Gaussian noise, and impulse noise. If salt-and-pepper noise is identified, adaptive median filtering is used for denoising. The filter window size of the adaptive median filter is adaptively adjusted from 3×3 to 7×7 according to the noise density. A 3×3 window is used when the noise density is ≤10%, and the window size is gradually expanded to 7×7 when the noise density is >10%. If Gaussian noise is identified, guided filtering is used for denoising. The guide image is selected from the image data, and the filter radius is... The regularization parameter is set to 5, and the regularization parameter is set to 1e-3 to ensure that edge details of the image data are preserved. If it is identified as impulse noise, a weighted nonlinear mean filter is used for denoising. The denoised image data is then filtered for abnormal images by the average local contrast and image entropy. The average local contrast is obtained by calculating the average difference between the gray values ​​of each pixel and its 8 neighboring pixels. The image entropy is calculated based on a 256-level gray level distribution. Finally, the denoised image data with an average local contrast ≥ 20 and an image entropy ≥ 4.5 is retained, resulting in the image data after noise anomaly processing.

[0014] Video data noise anomaly processing: For each frame of the video data, intra-frame denoising is first performed using 3D block matching filtering. The 3D block size of the 3D block matching filter is set to 8×8×3, the search window is set to 21×21×5, and similar 3D blocks are found through block matching and weighted average to achieve intra-frame denoising. For the intra-frame denoised video data, the Lucas-Kanade optical flow algorithm is used to calculate the motion vector between adjacent frames. The feature point selection of the Lucas-Kanade optical flow algorithm uses Shi-Tomasi corner detection, the optical flow calculation window size is set to 15×15, and the number of iterations is set to 10. The Euclidean distance of the optical flow vectors of the feature points of adjacent frames is calculated, and the average of all feature point distances is taken as the Euclidean distance of the motion vector between adjacent frames. When the Euclidean distance of the motion vector is greater than 5 pixels, it is determined that the next frame has motion disorder or redundancy. The previous frame image is retained and the next frame image is deleted. All adjacent frames are processed in sequence to obtain the video data after noise anomaly processing.

[0015] Noise and anomaly processing of text data: Special symbols in the text data are removed using regular expressions, retaining only letters, numbers, Chinese characters, spaces, and common Chinese punctuation marks; then, the text perplexity is calculated using a pre-trained BERT model. The text data after special symbols is uniformly truncated or padded with zeros to 128 characters as input to the BERT model. The text perplexity is calculated based on the probability prediction value of each character in the text sequence by the BERT model. Text data with a text perplexity ≤100 is retained, resulting in text data with noise and anomaly processing.

[0016] Audio data noise anomaly processing: Background noise in the original audio data is removed using spectral subtraction based on short-time Fourier transform (SFT). The frame length of the SFT is set to 20ms, the frame shift to 10ms, and the Hanning window is used as the windowing function. The audio signal-to-noise ratio (SNR) of each audio segment in the denoised audio data is calculated by the ratio of the signal power to the noise power of the audio segment. The signal power is taken as the average power of the audio segment, and the noise power is taken as the average noise power estimated by spectral subtraction. Audio segments with an SNR ≥ 20dB are retained. If the length of a single audio segment is less than 1 second, it is spliced ​​with an adjacent audio segment that meets the SNR requirement to a length of more than 1 second to obtain the noise anomaly processed audio data.

[0017] Preferably, the method for standardizing the format is as follows:

[0018] Standardization of video data format after noise anomaly processing: The noise-anomaly processed video data is uniformly divided along the time axis into a continuous frame sequence of 16 frames / segments. If the total number of frames of the noise-anomaly processed video data is not divisible by 16, the last frame segment with fewer than 16 frames is padded to 16 frames by inserting the last frame of the noise-anomaly processed video data at the end. A bilinear interpolation algorithm is used on each frame in the frame sequence to uniformly scale the resolution of each frame to 224×224 pixels, and at the same time, its color space is converted from the original format to RGB format, forming a frame sequence group stored in NumPy arrays.

[0019] Standardization of image data format after noise anomaly processing: The PIL library is used to adjust the resolution of the noise-anomaly processed image data. The BILINEAR interpolation method is used to uniformly adjust the resolution of the noise-anomaly processed image data to 224×224 pixels. If the noise-anomaly processed image data is a grayscale image or a single-channel image, it is converted to RGB three-channel format by channel copying. Then, based on the statistics of the ImageNet dataset, the pixel values ​​of the converted RGB three-channel format image data are standardized. Then, the standardized pixel values ​​are adjusted to the [0,1] interval by min-max mapping to obtain the image standard data stored in float32 type.

[0020] The text data format was standardized after noise anomaly processing: the jieba word segmentation library was used to perform word segmentation on the noise anomaly processed text data, and the Harbin Institute of Technology Chinese stop word list was called to traverse the word segmentation results and remove words belonging to the Harbin Institute of Technology Chinese stop word list; then, the BERT tokenizer matched with the text encoder was used to convert the noise anomaly processed text data after filtering stop words into word embedding sequences. The word embedding sequence was marked with [CLS] at the beginning and [SEP] at the end. If the word embedding sequence length exceeded 512, it was truncated to a length of 512 before the [SEP] mark; if the word embedding sequence length was less than 512, it was padded to a length of 512 with [PAD] mark after the [SEP] mark. The word embedding sequence was obtained accordingly.

[0021] The audio data that has been processed for noise abnormalities is resampled at a sampling rate of 16kHz, and spectral features are extracted through 40 Mel filters. At the same time, the time step of the spectral features is uniformly 128 steps by truncation or padding, and the Mel spectral features are obtained accordingly.

[0022] The audio data format after noise anomaly processing is standardized as follows: The Librosa audio processing library is used to perform a resampling operation on the noise anomaly processed audio data at a sampling rate of 16kHz. The resampling process uses a linear interpolation algorithm to ensure that the audio frequency features are not distorted. Then, based on the Mel frequency scale, 40 equally spaced Mel filters are used to extract the spectral features of the resampled audio data. At the same time, the time step of the spectral features is unified to 128 steps by truncating or padding zero values, resulting in Mel spectral features with a dimension of 40×128.

[0023] Preferably, the multimodal dataset includes frame sequence groups, image standard data, word embedding sequences, and Mel-spectral features; the modal encoder includes a video encoder, an image encoder, a text encoder, and an audio encoder.

[0024] The video encoder extracts video semantic features from video data by inputting frame sequence groups into the Video Swin Transformer-Tiny model.

[0025] The image encoder extracts the semantic features of the image data by inputting standard image data into a pre-trained visual model. The pre-trained visual model is ResNet-50, ViT-B / 16, or EfficientNet-B4.

[0026] The text encoder extracts the text semantic features of the text data by inputting the word embedding sequence into a pre-trained Transformer model. The pre-trained Transformer model is BERT-base, RoBERTa-base, or ALBERT-base-v2.

[0027] The audio encoder extracts audio semantic features from the audio data by inputting the Mel spectral features into the Audio Spectrogram Transformer-small model;

[0028] Preferably, the semantic features of each modality of data refer to the video semantic features of video data, the image semantic features of image data, the text semantic features of text data, and the audio semantic features of audio data, and the dimension of the semantic features of each modality of data is 768-dimensional.

[0029] Preferably, the process of normalizing and adapting the semantic features of each modality data is as follows:

[0030] Video semantic features, image semantic features, text semantic features, and audio semantic features are L2 normalized using the L2 normalization formula. Then, a modality adaptation layer is used to fine-tune the L2 normalized video semantic features, image semantic features, text semantic features, and audio semantic features to the feature distribution that meets the input requirements of the shared semantic space, resulting in a modality feature vector set including video feature vectors, image feature vectors, text feature vectors, and audio feature vectors.

[0031] The L2 normalization formula is as follows: ;

[0032] in, d represents the semantic features of each modality of data, and d is the dimension of the semantic features (768 dimensions). yes The feature value of the k-th dimension, These are the semantic features after L2 normalization, and the modulus of the normalized semantic features is always equal to 1.

[0033] The modality adaptation layer is a 1-layer fully connected network with ReLU activation function, and a BatchNorm2d layer is connected in series. The momentum parameter of the BatchNorm2d layer is set to 0.9 and the eps parameter is set to 1e-5. This is used to batch normalize the features output by the fully connected network to avoid subsequent mapping deviations caused by feature distribution shifts.

[0034] Preferably, the shared semantic space is constructed through a cross-modal projection network, which is a two-layer fully connected network, wherein the hidden layer dimension of the first fully connected network is set to 1024, the output layer dimension of the second fully connected network is set to 512, and both fully connected networks are followed by a ReLU activation function.

[0035] The modal feature vectors of each modal data are mapped to the shared semantic space through the mapping formula of the cross-modal projection network, and the initial shared feature vectors of each modal data are obtained at the same time.

[0036] The mapping formula for the cross-modal projection network is:

[0037] ;

[0038] in, It is the initial shared feature vector of each modality data mapped to the shared semantic space. These are the modal feature vectors of each modal data. It is the weight matrix of the first layer of the fully connected network. This is the first level bias term. It is the weight matrix of the second fully connected layer. It is the second-level bias term.

[0039] Preferably, the specific method for achieving semantic consistency alignment between modal feature vectors of different modal data corresponding to the same semantic content through the cross-modal alignment mechanism includes the following steps:

[0040] S31. Initialize the shared feature vector of the video data. Initial shared feature vector of image data Initial shared feature vectors of text data The initial shared feature vector of the audio data is The vectors are paired up to form 6 modality-shared feature vector pairs. The specific pairings are as follows: , ,

[0041] , , and ;

[0042] S32. Share feature vector pairs for each modality. (Where X and Y represent any two different modalities of data, and X,Y∈{V,I,T,A} and X≠Y) Attention weights are calculated using the dot product formula. and And the attention weight and The initial shared feature vectors of the two modal data in the feature vector pair shared with this modality are respectively and Weighted summation is performed to obtain the shared feature vector of each modality's data, i.e. , , and ;

[0043] The dot product formula is:

[0044] ;

[0045] ;

[0046] Where d is the dimension of the initial shared feature vector;

[0047] The formula for the weighted summation is:

[0048] ;

[0049] ;

[0050] S33. Replace the initial shared semantic space feature vector with the shared feature vector of each modality data, and re-execute S31. Repeat the iteration until the feature cosine similarity of any pairwise combination of the shared feature vectors of each modality data in the shared semantic space is ≥0.95, thus obtaining the shared feature vectors of each modality data with semantic consistency alignment, that is, obtaining the multimodal shared feature set. ).

[0051] Preferably, the intramodal enhancement method is as follows:

[0052] For the frame sequence group in the multimodal dataset, temporal adjustment is performed on each frame sequence, and the frame offset is controlled within ≤2 frames to avoid semantic gaps caused by frame order disorder. At the same time, spatial enhancement is performed by randomly cropping each frame image at a ratio of 0.8-1.0 to ensure that the key semantic regions of the core objects in the frame are preserved. Then, the brightness of the cropped frame image is adjusted by ±10% without changing the semantic discriminability of the frame. Finally, the video modality-enhanced frame sequence is obtained.

[0053] Gaussian blur with a standard deviation σ≤0.5 and pixel saturation adjustment with an amplitude of ±15% are applied to the standard image data in the multimodal dataset to obtain intramodal image enhancement data;

[0054] The standard image data in the multimodal dataset is blurred using a Gaussian filter with a standard deviation σ≤0.5 to avoid excessive blurring and loss of the image subject; then the saturation of its RGB three-channel pixels is adjusted by ±15% without changing the color attribute of the image subject, thus obtaining the image intramodal enhancement data;

[0055] For word embedding sequences in the multimodal dataset, based on the Harbin Institute of Technology Chinese thesaurus, ≤30% of non-stop words in the sequence are selected for synonym replacement. The replacement ratio is controlled within 30% to avoid semantic shift, thus obtaining the text modality-enhanced word embedding sequence.

[0056] The volume of the Mel spectral features in the multimodal dataset is fine-tuned by ±5% to avoid affecting the recognition of core content; the time step is stretched by 0.9-1.1 times, and after stretching, 128 time steps are restored by padding or truncation to maintain dimensionality uniformity; then weak background noise with a signal-to-noise ratio (SNR) ≥ 30dB is superimposed to avoid obscuring the core semantics of the audio, thus obtaining the audio modality-enhanced Mel spectral features. Based on this, intramodal enhanced data is formed, including video modality-enhanced frame sequences, image modality-enhanced data, text modality-enhanced word embedding sequences, and audio modality-enhanced Mel spectral features.

[0057] Preferably, the method for cross-modal enhancement is as follows:

[0058] Shared feature vectors from text data are input into the Stable Diffusion image generation model and the Tacotron2 audio generation model, respectively, to generate cross-modal augmented data for text and images and audio signals. Shared feature vectors from image data are input into the BERT-based pre-trained Transformer model and the VideoGPT video generation model, respectively, to generate cross-modal augmented frame sequences for image-text and image-video. Shared feature vectors from audio data are input into the Whisper speech recognition model and the Stable Diffusion image generation model, respectively, to generate cross-modal augmented data for audio-text and audio-image. Shared feature vectors from video data are input into the BART pre-trained model and the Tacotron2 audio generation model, respectively. 2. An audio generation model is used to generate video text and video-audio cross-modal augmented data. Then, noise anomaly processing and format standardization are performed on the audio signal, image text, audio text, and video text to obtain text-audio cross-modal augmented data, image-text cross-modal augmented data, audio-text cross-modal augmented data, and video-text cross-modal augmented data. Based on this, cross-modal augmented data including text-image cross-modal augmented data, image-video cross-modal augmented frame sequences, audio-image cross-modal augmented data, video-audio cross-modal augmented data, text-audio cross-modal augmented data, image-text cross-modal augmented data, audio-text cross-modal augmented data, and video-text cross-modal augmented data are formed.

[0059] Preferably, the method for integrating and deduplicating enhanced data is as follows:

[0060] The intramodal augmented data, cross-modal augmented data, and multimodal datasets are merged to form a deduplicated dataset. The shared feature vector corresponding to each data item in the deduplicated dataset is extracted. For data in the multimodal dataset, the corresponding shared feature vector in the multimodal shared feature set is directly called. For each data item in the augmented data, the corresponding modal feature vector is obtained through S2. The modal feature vector is then mapped through a cross-modal projection network to obtain the corresponding shared feature vector. The cosine similarity formula is used to calculate the similarity of the shared feature vectors of any two data items belonging to the same modality. When the similarity of the shared feature vectors of the two data items is ≥0.98, they are judged as duplicate data. One data item is retained and the other data item is removed to obtain the deduplicated dataset.

[0061] The deduplicated data set is categorized and linked by modality to form an alignment-enhanced multimodal dataset, which includes alignment-enhanced video data subsets, alignment-enhanced image data subsets, alignment-enhanced text data subsets, and alignment-enhanced audio data subsets.

[0062] Compared with the prior art, the present invention has the following beneficial effects:

[0063] 1. This invention overcomes the limitations of fragmented existing multimodal preprocessing and establishes a targeted denoising and unified standardized preprocessing system. Existing technologies for multimodal data preprocessing either only design denoising strategies for a single modality, lacking targeted processing for video and audio, or suffer from inconsistent format standards, exacerbating modal heterogeneity in subsequent feature extraction and alignment. This invention, however, addresses this by implementing targeted noise anomaly processing for different modalities. It designs differentiated solutions for the different noise characteristics of images, videos, text, and audio. Images are identified using noise detection algorithms to identify noise types and corresponding filtering methods are applied. Videos undergo intra-frame denoising, followed by optical flow algorithms to remove redundant frames. Audio segments are selected based on signal-to-noise ratio and stitched together when necessary. Simultaneously, it achieves unified format standardization across all modalities, uniformly adapting preprocessed data to the requirements of subsequent feature extraction and alignment, eliminating format barriers between different modalities, and laying the foundation for unified feature vector dimensions and shared semantic space mapping.

[0064] 2. This invention innovates a full-modal semantic alignment mechanism, upgrading from pairwise modal adaptation to full-modal semantic unification, effectively bridging the semantic gap. Existing technologies only design alignment strategies for pairwise modalities, requiring multiple transformations that lead to semantic loss, and rely on surface feature matching, making it difficult to handle semantic ambiguity. This invention constructs a unified shared semantic space and designs a cross-modal projection network to map the normalized feature vectors of each modality to a shared semantic space of the same dimension, making the semantic expressions of video, images, text, and audio comparable. Simultaneously, it employs an alignment logic of full-modal pairwise pairing, attention weight updates, and multiple rounds of iterative verification. First, the initial shared feature vectors of each modality are fully combined and paired without repetition. Attention weights are calculated using the dot product formula and used to update the shared feature vectors of each modality. Then, the updated feature vectors are used as the basis for repeated iterations until the feature cosine similarity of all pairwise modal combinations reaches a high consistency standard. This mechanism achieves full-modal alignment coverage in a single step, avoids semantic loss from multiple transformations, and strengthens semantically relevant region features through attention weights, effectively handling semantic ambiguity and significantly improving alignment accuracy.

[0065] 2. This invention constructs a collaborative enhancement system combining intra-modal and cross-modal approaches, balancing data diversity and semantic consistency. Existing technologies are either limited to intra-modal enhancement, failing to expand the multimodal data coverage dimension, or prone to semantic shifts in cross-modal enhancement, reducing model generalization ability. This invention employs parameter-controllable intra-modal enhancement, setting strict enhancement parameters for the semantic characteristics of each modality. For video, frame offsets are controlled to avoid action logic disruption; for images, moderate Gaussian blurring is used to preserve subject details; for text, the synonym replacement ratio is controlled to avoid semantic changes; and for audio, weak background noise is superimposed to avoid obscuring core content, ensuring semantic consistency between the enhanced and original data. Simultaneously, cross-modal enhancement driven by shared features uses semantic features from the multimodal shared feature set as semantic anchors to drive the generative model for cross-modal data generation. After generation, semantic matching is ensured through cosine similarity verification of shared features between the enhanced and original data and manual screening. This significantly increases the amount of multimodal data, and the semantic consistency rate of the enhanced data is at a high level, providing high-quality and highly diverse training data support for multimodal learning models.

[0066] 4. This invention establishes a unified deduplication standard based on semantic features, solving the problems of redundancy and quality control issues in augmented datasets. Existing technologies lack a unified logic for deduplication of augmented multimodal data; either they fail to identify semantically repetitive data based on surface features, or they fail to distinguish modality types, leading to chaotic deduplication results. This invention innovatively proposes a same-modality deduplication standard based on shared semantic features. It extracts the shared semantic features corresponding to the data to be deduplicated, sets a reasonable similarity threshold as the deduplication criterion, and only calculates similarity for data within the same modality, avoiding invalid comparisons between different modalities. This standard can effectively remove redundant samples from augmented data while retaining valid data that are semantically different but perform similarly, ensuring a high level of semantic overlap for repetitive data within the same modality. The resulting aligned augmented multimodal dataset combines low redundancy with high semantic consistency, significantly improving the training efficiency and generalization ability of multimodal learning models. Attached Figure Description

[0067] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 This is a flowchart of the method steps of the present invention. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be described in detail below. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other implementation methods obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0070] Examples, such as Figure 1 As shown, a multimodal data alignment and enhancement method based on semantic consistency includes the following steps:

[0071] S1. Obtain raw multimodal data containing video data, image data, text data, and audio data on the same semantic topic, perform noise anomaly processing and format standardization on the raw multimodal data to obtain a multimodal dataset;

[0072] S2. Extract the semantic features of each modality data from the multimodal dataset using a modal encoder, and normalize and adapt the semantic features of each modality data to obtain a modal feature vector set;

[0073] S3. Construct a shared semantic space, map the modal feature vectors of each modal data in the modal feature vector set to the shared semantic space, and achieve semantic consistency alignment between the modal feature vectors of each modal data corresponding to the same semantic content through a cross-modal alignment mechanism to obtain a multimodal shared feature set;

[0074] S4. Perform intra-modal augmentation on the modal data in the multimodal dataset and cross-modal augmentation on the shared feature vectors of the modal data in the multimodal shared feature set to obtain augmented data including intra-modal augmented data and cross-modal augmented data. Based on this, integrate and deduplicate the augmented data to obtain the aligned augmented multimodal dataset.

[0075] Furthermore, the working principle of the present invention will be illustrated below through embodiments:

[0076] This embodiment uses an "animal recognition" scenario as an example to describe in detail the specific implementation of the present invention. The multimodal data processed in this embodiment includes animal activity videos, animal static images, animal descriptive text, and animal call audio. The method of the present invention achieves semantic alignment and enhancement of the four types of modal data, which is ultimately applied to the training of an animal recognition model.

[0077] First, raw multimodal data was acquired, including 200 animal activity videos, 1000 animal still images, 500 animal descriptive texts, and 300 animal sound audio clips. The 200 animal activity videos included 10 types of animals such as cats, dogs, and birds, with each video lasting 5-10 seconds and in MP4 and AVI formats. The 1000 animal still images corresponded to the animal categories in the 200 animal activity videos, with resolutions ranging from 200×300 to 1080×960, and in JPG and PNG formats. The data consisted of 300 audio clips of animal sounds, including barks from dogs, meows from cats, and birdsong from 10 different animals, in WAV and MP3 formats, with durations ranging from 2 to 5 seconds. Noise and outlier processing was then performed. For video data, 3D block-matching filtering was used to denoise each frame. The Lucas-Kanade optical flow algorithm was used to calculate motion vectors between adjacent frames, eliminating redundant frames with excessively large Euclidean distances in the motion vectors, and retaining video segments with coherent motion. For image data, a noise detection algorithm was used to identify salt-and-pepper noise and Gaussian noise, and adaptive noise was applied accordingly. Median filtering and guided filtering were used for noise reduction, selecting clear images with an average local contrast ratio ≥20 and an image entropy value ≥4.5. For text data, regular expressions were used to remove special symbols, retaining letters, numbers, Chinese characters, and common punctuation. A pre-trained BERT model was used to calculate text perplexity and eliminate semantically confusing text. For audio data, spectral subtraction was used to remove background noise, and the audio signal-to-noise ratio (SNR) was calculated, retaining effective segments with an SNR ≥20dB. Segments shorter than 1 second were stitched together to complete the segment. Finally, format standardization was performed: video data was decomposed into 16-frame / segment sequences, and bilinear interpolation was used to unify the resolution to 224×224, converting it to RGB format to form a 16×224×224×3 NumPy array. Image data was adjusted to 224×224 resolution using BILINEAR interpolation, and grayscale images were converted to RGB three channels and normalized to the [0,1] interval based on ImageNet statistics. For text data, Chinese was segmented using jieba, and English was segmented using NLTK. After removing stop words, BERT was used for further processing. The tokenizer converts the data into a 512-length word embedding sequence. The audio data is resampled at 16kHz, and 40-dimensional Mel filter bank features are extracted. The time step is adjusted to 128 to form 40×128 Mel spectral features. The final output is a preprocessed multimodal dataset containing video frame sequence groups, image standard data, text word embedding sequences, and audio Mel spectral features.

[0078] Semantic features are extracted using a modal encoder. Video semantic features are extracted using the Video SwinTransformer-Tiny model, which takes a sequence of video frames as input and extracts spatiotemporal semantic features, outputting 768-dimensional video semantic features. Image semantic features are extracted using the ResNet-50 model, which takes 224×224 standard image data as input and extracts spatial semantic features, outputting 768-dimensional image semantic features. Text semantic features are extracted using the bert-base-chinese model, using the [CLS] token to output text semantic features. Audio semantic features are extracted using the Audio Spectrogram Transformer-small model, which takes Mel spectrum features as input and extracts frequency and rhythm semantics, outputting audio semantic features. Normalization and adaptation are then performed. L2 normalization is applied to the semantic features of all modal data to create modal feature vectors. A modal adaptation layer, consisting of a fully connected network and a BatchNorm2d layer, is then used to fine-tune the feature distribution to meet the input requirements of the shared semantic space. Finally, a normalized and adapted set of modal feature vectors is output, containing video, image, text, and audio feature vectors, all of which are 768-dimensional.

[0079] A cross-modal projection network is designed to map modal feature vectors to a 512-dimensional shared semantic space using a cross-modal projection network mapping formula, simultaneously obtaining initial shared feature vectors for each modality. Subsequently, a cross-modal alignment mechanism is used to process the data, first forming six sets of modal shared feature vector pairs (video-image, video-text, video-audio, image-text, image-audio, and text-audio). Attention weights are calculated for each set of modal shared feature vector pairs using a dot product formula. Then, a weighted summation formula is used to update the shared feature vectors for the X modal features under the same semantic topic. This process is repeated until the feature cosine similarity of all modal shared feature vector pairs is ≥0.95, at which point the iteration stops, and a multimodal shared feature set is output.

[0080] Intramodal enhancement is performed on the multimodal dataset. Frame sequences in the frame sequence group undergo temporal adjustment with a frame offset ≤2, random cropping at a ratio of 0.8-1.0, and ±10% brightness adjustment. Standard image data undergoes Gaussian blur with σ≤0.5, ±15% color jitter, and ≤10% random background erasure. Word embedding sequences are replaced with ≤30% of non-stop words based on a thesaurus of synonyms, such as "running" → "running". Mel-spectral features undergo ±5% volume fine-tuning, 0.9-1.1x time stretching, and the addition of environmental noise with an SNR ≥30dB, such as a slight wind sound. Subsequently, cross-modal enhancement is performed on the shared feature vectors of each modality in the multimodal shared feature set. The shared feature vectors of text data are enhanced using a Stable... Diffusion generates animal images, Tacotron2 generates corresponding audio calls, and shared feature vectors from image data are used to generate animal description text via BART and 3-second animal motion videos via VideoGPT. Shared feature vectors from audio data are used to convert calls to text via Whisper and then generate corresponding animal images. Shared feature vectors from video data are used to generate video description text via BART and generate corresponding animal audio calls. Finally, enhancement validation and deduplication are performed. Intramodal enhancement data, cross-modal enhancement data, and multimodal datasets are merged. The similarity of shared feature vectors of two data points within the same modality is calculated. If the similarity of shared feature vectors of two data points is ≥0.98, they are considered duplicate data. One data point is retained, and the other is removed to obtain a deduplicated dataset. The deduplicated dataset is then categorized and associated by modality to form an aligned and enhanced multimodal dataset containing subsets of aligned and enhanced video data, aligned and enhanced image data, aligned and enhanced text data, and aligned and enhanced audio data.

[0081] After processing using the obtained aligned and enhanced multimodal dataset in this embodiment, the animal recognition model was trained, and the training effect was significantly improved: the cross-modal retrieval accuracy increased from 62% to 88%, verifying the effectiveness of the method of the present invention in multimodal data alignment and enhancement.

[0082] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them; modifications to the technical solutions described in the foregoing embodiments, or equivalent substitutions of some of the technical features, do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A multimodal data alignment and enhancement method based on semantic consistency, characterized in that, Includes the following steps: S1. Obtain raw multimodal data containing video data, image data, text data, and audio data on the same semantic topic, perform noise anomaly processing and format standardization on the raw multimodal data to obtain a multimodal dataset; S2. Extract the semantic features of each modality data from the multimodal dataset using a modal encoder, and normalize and adapt the semantic features of each modality data to obtain a modal feature vector set; S3. Construct a shared semantic space, map the modal feature vectors of each modal data in the modal feature vector set to the shared semantic space, and achieve semantic consistency alignment between the modal feature vectors of each modal data corresponding to the same semantic content through a cross-modal alignment mechanism to obtain a multimodal shared feature set; S4. Perform intra-modal augmentation on the modal data in the multimodal dataset and cross-modal augmentation on the shared feature vectors of the modal data in the multimodal shared feature set to obtain augmented data including intra-modal augmented data and cross-modal augmented data. Based on this, integrate and deduplicate the augmented data to obtain the aligned augmented multimodal dataset.

2. The multimodal data alignment and enhancement method based on semantic consistency according to claim 1, characterized in that, The method for noise anomaly processing of the original multimodal data is as follows: The image data is denoised by identifying the noise type using a noise detection algorithm. If the noise type is salt-and-pepper noise, adaptive median filtering is used for denoising. If the noise type is Gaussian noise, guided filtering is used for denoising. If the noise type is impulse noise, nonlinear mean filtering is used for denoising. The denoised image data is then filtered for abnormal images by using the average local contrast and image entropy value. The denoised image data with an average local contrast ≥20 and an image entropy value ≥4.5 are retained to obtain the image data after noise anomaly processing. After intra-frame denoising is performed on each frame of the video data using 3D block matching filtering, the motion vector between adjacent frames is calculated using the Lucas-Kanade optical flow algorithm. When the Euclidean distance of the motion vector is greater than 5 pixels, the previous frame is retained and the next frame is deleted, thereby obtaining the video data after noise anomaly processing. Special characters are removed from the text data using regular expressions, and the text perplexity is calculated using the BERT model. Text data with a perplexity ≤100 is retained, thus obtaining text data that has been processed for noise anomalies. Background noise is removed from the audio data using spectral subtraction, and the audio signal-to-noise ratio of the audio segments in the audio data is calculated. Audio segments with an audio signal-to-noise ratio ≥20dB are retained to obtain audio data that has undergone noise anomaly processing.

3. The multimodal data alignment and enhancement method based on semantic consistency according to claim 2, characterized in that, The method for format standardization is as follows: The video data that has been processed for noise abnormalities is broken down into a frame sequence of 16 frames / segments. Each frame in the frame sequence is uniformly scaled to a resolution of 224×224 and converted to RGB format to form a frame sequence group. The image data that has undergone noise anomaly processing is adjusted to a resolution of 224×224 using the PIL library and converted to RGB three-channel format. At the same time, the pixel values ​​of the converted RGB three-channel image data are normalized to the [0,1] interval using ImageNet statistics to obtain standard image data. The text data that has been processed for noise anomalies is segmented using jieba, and after filtering stop words using the Chinese stop word list of Harbin Institute of Technology, it is converted into a word embedding sequence of length 512 using BERT tokenizer; The audio data that has been processed for noise abnormalities is resampled at a sampling rate of 16kHz, and spectral features are extracted through 40 Mel filters. At the same time, the time step of the spectral features is uniformly 128 steps by truncation or padding, and the Mel spectral features are obtained accordingly.

4. The multimodal data alignment and enhancement method based on semantic consistency according to claim 3, characterized in that, The multimodal dataset includes frame sequence groups, standard image data, word embedding sequences, and Mel-spectral features; the modal encoders include video encoders, image encoders, text encoders, and audio encoders. The video encoder extracts video semantic features from video data by inputting frame sequence groups into the Video Swin Transformer-Tiny model. The image encoder extracts the semantic features of the image data by inputting standard image data into a pre-trained visual model. The pre-trained visual model is ResNet-50, ViT-B / 16, or EfficientNet-B4. The text encoder extracts the text semantic features of the text data by inputting the word embedding sequence into a pre-trained Transformer model. The pre-trained Transformer model is BERT-base, RoBERTa-base, or ALBERT-base-v2. The audio encoder extracts audio semantic features from the audio data by inputting the Mel spectral features into the Audio Spectrogram Transformer-small model.

5. The multimodal data alignment and enhancement method based on semantic consistency according to claim 4, characterized in that, The process of normalizing and adapting the semantic features of each modality data is as follows: Video semantic features, image semantic features, text semantic features, and audio semantic features are L2 normalized using the L2 normalization formula. Then, a modality adaptation layer is used to fine-tune the L2 normalized video semantic features, image semantic features, text semantic features, and audio semantic features to the feature distribution that meets the input requirements of the shared semantic space, resulting in a modality feature vector set including video feature vectors, image feature vectors, text feature vectors, and audio feature vectors. The modality adaptation layer is a 1-layer fully connected network with ReLU activation function, and is connected in series with a BatchNorm2d layer.

6. The multimodal data alignment and enhancement method based on semantic consistency according to claim 1, characterized in that, The shared semantic space is constructed through a cross-modal projection network, which is a two-layer fully connected network. The hidden layer dimension of the first fully connected network is set to 1024, and the output layer dimension of the second fully connected network is set to 512. Both fully connected networks are followed by a ReLU activation function. The modal feature vectors of each modal data are mapped to the shared semantic space through the mapping formula of the cross-modal projection network, and the initial shared feature vectors of each modal data are obtained at the same time.

7. The multimodal data alignment and enhancement method based on semantic consistency according to claim 6, characterized in that, The method for achieving semantic consistency alignment between modal feature vectors of different modal data corresponding to the same semantic content through the cross-modal alignment mechanism includes the following steps: S31. Pair the initial shared feature vectors of each modality data into pairs without repetition to obtain several modality shared feature vector pairs; S32. For each modality shared feature vector pair, the attention weight is calculated using the dot product formula, and the attention weight is weighted and summed with the initial shared feature vectors of the two modal data in the modality shared feature vector pair, thereby obtaining the shared feature vector of each modality data; S33. Replace the initial shared semantic space feature vector with the shared feature vector of each modality data, re-execute S31, and repeat the iteration until the feature cosine similarity of any pairwise combination of the shared feature vectors of each modality data in the shared semantic space is ≥0.95, thus obtaining the shared feature vectors of each modality data with semantic consistency alignment.

8. The multimodal data alignment and enhancement method based on semantic consistency according to claim 7, characterized in that, The method for intramodal enhancement is as follows: Perform timing adjustments with a frame offset ≤2, random cropping with a ratio of 0.8-1.0, and brightness adjustments with an amplitude of ±10% on the frame sequences in the frame sequence group to obtain the video modality-enhanced frame sequence; Gaussian blur with a standard deviation σ≤0.5 and pixel saturation adjustment with an amplitude of ±15% are applied to the standard image data to obtain intramodal image enhancement data; Perform synonym forest replacement on the word embedding sequence to replace ≤30% of non-stop words, resulting in an enhanced word embedding sequence within the text modality; By performing volume fine-tuning of ±5% on the Mel spectral features, stretching by 0.9-1.1 times the time step, and superimposing weak background noise with a signal-to-noise ratio (SNR) ≥ 30 dB, the enhanced Mel spectral features within the audio modality are obtained.

9. The multimodal data alignment and enhancement method based on semantic consistency according to claim 8, characterized in that, The method for cross-modal enhancement is as follows: The shared feature vectors of text data are input into the Stable Diffusion image generation model and the Tacotron 2 audio generation model, respectively, to generate cross-modal augmented data of text and image and audio signals. The shared feature vectors of image data are input into the BERT-based pre-trained Transformer model and the VideoGPT video generation model, respectively, to generate cross-modal augmented frame sequences of image-text and image-video. The shared feature vectors of audio data are input into the Whisper speech recognition model and the Stable Diffusion image generation model, respectively, to generate cross-modal augmented data of audio-text and audio-image. The shared feature vectors of video data are input into the BART pre-trained model and the Tacotron 2 audio generation model, respectively, to generate cross-modal augmented data of video-text and video-audio. Then, noise anomaly processing and format standardization are performed on the audio signals, image text, audio text, and video text to obtain cross-modal augmented data of text-audio, image-text, audio-text, and video-text.

10. The multimodal data alignment and enhancement method based on semantic consistency according to claim 9, characterized in that, The method for enhancing data integration and deduplication is as follows: The intramodal augmentation data, cross-modal augmentation data, and multimodal datasets are merged to form a dataset to be deduplicated. The shared feature vector corresponding to each data item in the dataset to be deduplicated is extracted. The cosine similarity formula is used to calculate the similarity of the shared feature vectors of any two data items belonging to the same modality. When the similarity of the shared feature vectors of the two data items is ≥0.98, they are judged as duplicate data. One data item is retained and the other data item is removed to obtain the deduplicated dataset. The deduplicated data set is categorized and linked by modality to form an alignment-enhanced multimodal dataset, which includes alignment-enhanced video data subsets, alignment-enhanced image data subsets, alignment-enhanced text data subsets, and alignment-enhanced audio data subsets.

Images