Large model cross-modal collaborative understanding method and device

Abstract

The invention provides a large-model cross-modal collaborative understanding method and device. The fusion efficiency and the understanding capability of multi-modal information can be improved. The large-model cross-modal collaborative understanding method comprises the following steps: preprocessing visual data, language data and sound data to obtain preprocessed visual data, language data and sound data; wherein the preprocessing comprises the steps of performing adaptive size adjustment and normalization on visual data, performing word segmentation and dynamic truncation on language data, and performing band-pass filtering and spectral noise reduction on sound data; extracting visual features, language features and sound features based on the preprocessed visual data, language data and sound data; on the basis of an adaptive mapping network and a mixed granularity cross-modal attention mechanism, performing feature alignment on the visual features, the language features and the sound features to obtain aligned feature vectors; and based on a dynamic routing architecture, fusing the aligned feature vectors to generate a unified multi-modal representation.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and apparatus for cross-modal collaborative understanding of large models. Background Technology

[0002] With the rapid development of artificial intelligence technology, single-modal perception and understanding can no longer meet the complex and ever-changing information processing needs of the real world. Multimodal learning, as a technology that integrates multiple perceptual information, has gradually become a research hotspot. Currently, large-scale multimodal models mainly focus on the fusion of vision and language, and have achieved certain results. In the field of speech processing, the WaveNet model directly generates high-quality speech signals through autoregressive neural networks, significantly improving the naturalness of speech synthesis. However, existing multimodal models still face challenges in integrating visual, language, and auditory information. For example, data from different modalities differ in feature space, making direct joint modeling difficult and affecting the model's generalization ability and inference efficiency. Summary of the Invention

[0003] To address the aforementioned technical problems, this invention is proposed. Embodiments of this invention provide a method and apparatus for cross-modal collaborative understanding of large models, which can improve the fusion efficiency and understanding capability of multimodal information.

[0004] According to one aspect of the present invention, a method for cross-modal collaborative understanding of large models is provided, comprising: preprocessing visual data, language data, and sound data to obtain preprocessed visual data, language data, and sound data; wherein the preprocessing includes: adaptively adjusting and normalizing the visual data, segmenting and dynamically truncating the language data, and performing bandpass filtering and spectral subtraction noise reduction on the sound data; extracting visual features, language features, and sound features based on the preprocessed visual data, language data, and sound data; aligning the visual features, language features, and sound features based on an adaptive mapping network and a hybrid granularity cross-modal attention mechanism to obtain aligned feature vectors; and fusing the aligned feature vectors based on a dynamic routing architecture to generate a unified multimodal representation.

[0005] In one embodiment, the visual data includes image data and video data. Adaptive size adjustment and normalization of the visual data includes: for image data, using an adaptive size adjustment algorithm to unify the resolution to a preset size while maintaining the key features of the image data; for video data, extracting key frames, using a motion detection algorithm based on optical flow to filter frames with dynamic changes greater than a preset change value, and converting the continuous video stream into a preset number of frame sequences.

[0006] In one embodiment, the language data includes Chinese data and English data. The language data is segmented and dynamically truncated, including: segmenting the Chinese and English data; wherein the Chinese data is segmented using the jieba tool library, and the English data is segmented using the NLTK tool library; when the length of the language data exceeds a preset length, a sliding window truncation strategy, combined with a preset maximum input length limit, is used to dynamically truncate and retain key semantic segments; and a spelling correction module based on language model perplexity performs spelling correction on the segmented Chinese and English data.

[0007] In one embodiment, bandpass filtering and spectral subtraction are performed on the audio data to reduce noise, including: performing a multi-stage noise reduction process on the audio data; wherein the multi-stage noise reduction process includes removing environmental noise frequency bands through a bandpass filter, separating speech segments and silence segments using speech activity detection technology based on a hidden Markov model, and reducing noise in the speech segments using spectral subtraction or a deep learning noise reduction model; wherein preprocessing the visual data, language data, and audio data to obtain the preprocessed visual data, language data, and audio data further includes: converting the audio data from different sources into a target sampling rate; performing endpoint detection and prosodic feature extraction on the audio data to generate Mel spectrograms or linearly predicted cepstral coefficients.

[0008] In one embodiment, based on the preprocessed visual data, language data, and sound data, visual features, language features, and sound features are extracted, including: constructing a three-level feature extraction system to extract the visual data, language data, and sound data into abstract semantic visual features, language features, and sound features; wherein, the three-level feature extraction system includes: basic feature extraction, task adaptation fine-tuning, and cross-modal enhancement preprocessing.

[0009] In one embodiment, based on an adaptive mapping network and a hybrid granularity cross-modal attention mechanism, the visual features, language features, and sound features are aligned to obtain aligned feature vectors. This includes: based on the adaptive mapping network, performing dimensional matching on the visual features, language features, and sound features of different dimensions through basic dimensional alignment, semantic space calibration, and cross-modal interaction enhancement to obtain feature vectors, and introducing a modality-specific parameter matrix to ensure that different modalities retain unique feature distributions in a shared space; constraining the consistency of the mapped feature distributions through cross-modal contrastive loss to ensure that the semantic similarity of the aligned feature vectors is greater than a preset similarity threshold; and constructing a hybrid granularity attention model that integrates global contextual attention and local region attention.

[0010] In one embodiment, a hybrid granularity attention model is constructed, which integrates global contextual attention and local region attention. The model includes: calculating the global correlation of feature sequences of different modalities to generate cross-modal context vectors; wherein the modalities include visual, language, and sound; and constructing a position-sensitive attention mechanism based on the spatial locality of the visual features and the temporal locality of the language features and the sound features, by incorporating relative position encoding when calculating attention weights to preserve structural information within the modality.

[0011] In one embodiment, based on a dynamic routing architecture, aligned feature vectors are fused to generate a unified multimodal representation, including: calculating the second-order correlation between modalities using a bilinear interaction tensor; wherein the modalities include vision, language, and sound; fusing the aligned feature vectors based on the second-order correlation between modalities and the aligned feature vectors; and generating a unified multimodal representation by using the fused feature vectors as input through an autoregressive method based on a Transformer decoder architecture.

[0012] In one embodiment, the large-model cross-modal collaborative understanding method further includes: converting the unified multimodal representation into a task-preset output based on a task-adaptive processing framework; wherein the task-adaptive processing framework includes a classification task, a generation task, and a retrieval task; during the classification task processing, a multilayer perceptron classification head is accessed, and the feature weights of vision, language, and sound in the classification decision are configured based on preset weight parameters; during the generation task processing, the fused unified multimodal representation is used as a context vector, and modality type information is fused through position encoding; during the retrieval task processing, a query modality encoder and a database modality encoder are constructed, and the semantic distance between modalities is optimized through a comparative loss function.

[0013] According to another aspect of the present invention, a method apparatus for cross-modal collaborative understanding of large models is provided, comprising: a preprocessing module for preprocessing visual data, language data, and sound data to obtain preprocessed visual data, language data, and sound data; wherein the preprocessing includes: adaptively adjusting and normalizing the visual data, segmenting and dynamically truncating the language data, and performing bandpass filtering and spectral subtraction noise reduction on the sound data; a feature extraction module for extracting visual features, language features, and sound features based on the preprocessed visual data, language data, and sound data; an alignment module for aligning the visual features, language features, and sound features based on an adaptive mapping network and a hybrid granularity cross-modal attention mechanism to obtain aligned feature vectors; and a fusion module for fusing the aligned feature vectors based on a dynamic routing architecture to generate a unified multimodal representation.

[0014] The large-scale model cross-modal collaborative understanding method and apparatus provided by this invention features a refined processing flow designed for the characteristics of different modal data. This ensures that the basic data input to the model possesses high quality and consistency standards between modalities. Through multi-level alignment and a hybrid granularity attention mechanism, fine-grained semantic alignment and deep interaction are achieved between visual, linguistic, and auditory modalities. An adaptive mapping network is constructed to map heterogeneous modal features to the same representation space. Therefore, by effectively fusing information from visual, linguistic, and auditory modalities, a comprehensive understanding of complex scenes is achieved, enhancing the model's perception and reasoning capabilities. This improves the efficiency and understanding of multimodal information fusion, addressing the problems of isolated information and poor fusion effects between modalities in existing technologies. Attached Figure Description

[0015] The above and other objects, features, and advantages of the present invention will become more apparent from the more detailed description of the embodiments of the invention in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same parts or steps.

[0016] Figure 1 This is a flowchart illustrating a large-model cross-modal collaborative understanding method provided in an exemplary embodiment of the present invention.

[0017] Figure 2 This is a schematic diagram of the structure of a large model cross-modal collaborative understanding device provided in an exemplary embodiment of the present invention. Detailed Implementation

[0018] Hereinafter, exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments of the present invention. It should be understood that the present invention is not limited to the exemplary embodiments described herein.

[0019] With the rapid development of artificial intelligence technology, single-modal perception and understanding can no longer meet the complex and ever-changing information processing needs of the real world. Multimodal learning, as a technology that integrates multiple perceptual information, has gradually become a research hotspot. Currently, large-scale multimodal models mainly focus on the fusion of vision and language, and some progress has been made.

[0020] In the field of speech processing, the WaveNet model directly generates high-quality speech signals through an autoregressive neural network, significantly improving the naturalness of speech synthesis. However, existing multimodal models still face challenges in fusing visual, linguistic, and auditory information, mainly in the following aspects:

[0021] (1) Insufficient intermodal information fusion: Existing models mostly use simple feature splicing or attention mechanisms for modal fusion, lacking deep semantic alignment and interaction, resulting in limited fusion effect.

[0022] (2) Lack of a unified representation space: Data from different modalities differ in feature space, making it difficult to perform joint modeling directly, which affects the generalization ability and inference efficiency of the model.

[0023] (3) High training and inference costs: Multimodal models usually have a large number of parameters, and the training and inference processes consume a lot of computational resources, which limits their deployment and promotion in practical applications.

[0024] Therefore, this application proposes a method and apparatus for cross-modal collaborative understanding of large models, which can effectively integrate visual, linguistic and auditory information to achieve unified representation learning and efficient collaborative understanding, thereby improving the application performance of multimodal large models in complex scenarios.

[0025] Figure 1 This is a flowchart illustrating a large-model cross-modal collaborative understanding method provided in an exemplary embodiment of the present invention, as shown below. Figure 1 As shown, according to one aspect of the present invention, a method for cross-modal collaborative understanding of large models is provided, comprising: preprocessing visual data, language data, and sound data to obtain preprocessed visual data, language data, and sound data (see...). Figure 1 S110); wherein, preprocessing includes: adaptive size adjustment and normalization of visual data, word segmentation and dynamic truncation of language data, bandpass filtering and spectral subtraction noise reduction of audio data; based on the preprocessed visual data, language data and audio data, visual features, language features and audio features are extracted (see S110); Figure 1 S120); Based on an adaptive mapping network and a hybrid granularity cross-modal attention mechanism, visual features, language features, and sound features are aligned to obtain aligned feature vectors (see S120). Figure 1 S130); Based on a dynamic routing architecture, aligned feature vectors are fused to generate a unified multimodal representation (see S130); Figure 1 (S140).

[0026] In some embodiments of S110, the visual data includes image data and video data. For image data, an adaptive size adjustment algorithm is used to unify the resolution to a preset size while maintaining the key features of the image data. For video data, key frame extraction is performed, and a motion detection algorithm based on optical flow is used to filter frames with dynamic changes greater than a preset change value, converting the continuous video stream into a preset number of frame sequences.

[0027] As one possible implementation of S110, for image data, an adaptive resizing algorithm is first employed. Based on the target model input requirements, methods such as bilinear interpolation, nearest neighbor interpolation, or Lanczos interpolation are dynamically selected to maintain key image features while standardizing the resolution to a fixed size (e.g., 224×224 or 384×384). Next, in the normalization stage, in addition to traditional mean-variance normalization (subtracting the ImageNet dataset mean from each of the RGB channels and dividing by the standard deviation), a grayscale normalization algorithm for special scenarios such as medical images is introduced, improving contrast through histogram equalization.

[0028] As another possible implementation of S110, during video data processing, keyframe extraction is first performed, and a motion detection algorithm based on optical flow is used to filter frames with significant dynamic changes. Then, combined with a time series sampling strategy (such as uniform sampling or adaptive sampling based on attention mechanism), the continuous video stream is converted into a fixed number of frame sequences. Finally, the processing flow for each frame is consistent with the adaptive size adjustment flow for image data.

[0029] In some embodiments of S110, the language data includes Chinese data and English data. Segmenting and dynamically truncating the language data may include: segmenting the Chinese and English data; wherein the Chinese data is segmented using the jieba tool library, and the English data is segmented using the NLTK tool library; when the length of the language data exceeds a preset length, a sliding window truncation strategy, combined with a preset maximum input length limit, is used to dynamically truncate and retain key semantic segments; and a spelling correction module based on the perplexity of the language model performs spelling correction on the segmented Chinese and English data.

[0030] As a possible implementation of S110, it can integrate multilingual word segmentation engines, supporting jieba segmentation for Chinese, NLTK segmentation for English, and cross-language SentencePiece segmentation. Customized stop word lists (including general stop words and domain-specific stop words) can be used for different language characteristics. For long text processing, a sliding window truncation strategy is introduced, combined with the maximum input length limit of models such as BERT (e.g., 512 tokens), to dynamically truncate and preserve key semantic fragments.

[0031] As another possible implementation of S110, domain-specific cleaning rules can be designed for special scenarios such as code text and social media text, such as removing code comments and parsing the semantics of emojis.

[0032] In some embodiments of S110, a spelling correction module can also be introduced in the post-segmentation processing stage to automatically correct spelling errors based on the edit distance algorithm and language model perplexity calculation.

[0033] In some embodiments of S110, bandpass filtering and spectral subtraction for noise reduction of sound data may include: performing a multi-stage noise reduction process on the sound data; wherein, the multi-stage noise reduction process includes removing environmental noise frequency bands through a bandpass filter, separating speech segments and silence segments using speech activity detection technology based on a hidden Markov model, and using spectral subtraction or a deep learning noise reduction model to reduce noise in the speech segments; wherein, preprocessing the visual data, language data, and sound data to obtain preprocessed visual data, language data, and sound data further includes: converting sound data from different sources into a target sampling rate; performing endpoint detection and prosodic feature extraction on the sound data to generate Mel spectrograms or linearly predicted cepstral coefficients.

[0034] As one possible implementation of S110, audio preprocessing includes a multi-stage noise reduction process: first, ambient noise frequency bands are removed by bandpass filters; then, speech activity detection (VAD) technology based on Hidden Markov Model (HMM) is used to separate speech segments from silence segments; and finally, spectral subtraction or deep learning noise reduction models (such as Wave-U-Net) are used for refined noise reduction.

[0035] As one possible implementation of S110, the sampling rate unification stage uses resampling technology to convert audio from different sources (44.1kHz, 16kHz, etc.) to the target sampling rate (e.g., 16kHz), and combines it with an anti-aliasing filter to avoid frequency distortion.

[0036] As a possible implementation of S110, for speech data, additional endpoint detection and prosodic feature extraction are performed to generate Mel-Spectrogram or Linear Predicted Cepstral Coefficients (LPCC) as input for subsequent processing, while also supporting direct input of raw waveform data to the end-to-end model.

[0037] In some embodiments of S120, a three-level feature extraction system is constructed to extract visual data, language data, and sound data into abstract semantic visual features, language features, and sound features; wherein, the three-level feature extraction system includes: basic feature extraction, task adaptation fine-tuning, and cross-modal enhancement preprocessing.

[0038] As a possible implementation of S120, for feature extraction of visual data, an improved Visual Transformer (ViT) architecture can be adopted. Based on the standard ViT's 16×16 image blocks, a multi-scale block strategy is introduced (e.g., simultaneously processing 8×8, 16×16, and 32×32 blocks), dynamically fusing features at different scales through attention weights. In the pre-training phase, a hybrid dataset (including ImageNet, COCO, ADE20K, etc.) is used, employing contrastive learning and masked image modeling (MIM) for joint training, with the mask ratio dynamically adjusted (10%-50%) to enhance model robustness. For video processing scenarios, a temporal self-attention layer is added to the spatial attention module to construct a spatiotemporal Transformer (VideoViT), fusing inter-frame temporal information through 3D positional encoding. The fine-tuning phase supports two modes: lightweight fine-tuning by freezing the backbone network and training only the classification head, or full network fine-tuning combined with task-specific data augmentation (e.g., random erasure, color jitter), adapting to different task requirements such as image classification, object detection, and video action recognition.

[0039] As a possible implementation of S120, for feature extraction of language data, a deep bidirectional pre-trained RoBERTa variant can be used. This variant optimizes the positional encoding mechanism for long text processing (using relative positional encoding instead of absolute positional encoding) and introduces a dynamic masking strategy (generating different token masks when masking the same text multiple times). The word segmentation module supports mixed input of subword units and character-level representations, dynamically selecting the optimal representation through a gating mechanism. For domain adaptation, a multi-domain pre-trained corpus covering general domains (Wikipedia) and specialized domains (medical literature, legal texts) is constructed. During fine-tuning, an adapter technique is used, training only a small number of task-specific parameters while freezing the core parameters, significantly reducing fine-tuning costs. For cross-language tasks, the model is extended to a multilingual version of mBERT, aligning the semantic spaces of different languages ​​through adversarial training, supporting zero-shot transfer across multiple languages.

[0040] As a possible implementation of S120, feature extraction of audio data can be improved based on the Wav2Vec2.0 architecture. On top of the original self-supervised pre-training task (contrastive prediction coding CPC), a pseudo-label-based semi-supervised training stage can be added, utilizing a small amount of labeled data to enhance feature discrimination capabilities. Audio data feature extraction can be divided into two stages: first, acoustic feature maps are generated through a multi-layer convolutional neural network, and then input into a Transformer encoder for contextual modeling. For speech recognition tasks, a Connectionist Temporal Classification (CTC) decoder or an attention mechanism decoder can be added after the pre-trained model; for tasks such as emotion recognition, a temporal pooling module (such as bidirectional LSTM combined with attention pooling) can be added to the feature output layer. Furthermore, a cross-modal pre-training task can be set up to align speech features with visual lip movement data, enhancing the model's ability to capture speech-visual correlation information.

[0041] In some embodiments of S130, based on an adaptive mapping network, visual features, language features, and sound features of different dimensions are matched to obtain feature vectors through basic dimension alignment, semantic space calibration, and cross-modal interaction enhancement. A modality-specific parameter matrix is ​​introduced to ensure that different modalities retain unique feature distributions in a shared space. The consistency of the mapped feature distribution is constrained by cross-modal contrastive loss to ensure that the semantic similarity of the aligned feature vectors is greater than a preset similarity threshold. A hybrid granularity attention model is constructed to integrate global contextual attention and local region attention.

[0042] In some embodiments of S130, a hybrid granularity attention model is constructed, which integrates global contextual attention and local region attention. This includes: calculating the global correlation of feature sequences of different modalities and generating cross-modal context vectors; wherein the modalities include vision, language and sound; and constructing a position-sensitive attention mechanism to address the spatial locality of visual features and the temporal locality of language and sound features, by incorporating relative position encoding when calculating attention weights to preserve structural information within the modality.

[0043] As a possible implementation of S130, an adaptive mapping network is designed for feature mapping and dimensionality unification. For the initial dimensions of different modal features (e.g., 1024 dimensions for visual features, 768 dimensions for language features, and 512 dimensions for sound features), a dynamic dimensionality matching algorithm is employed. Specifically, for modalities with small dimensional differences (such as language and sound), a single-layer linear transformation combined with Layer Normalization is used; for high-dimensional differences between vision and other modalities, a multilayer perceptron (MLP) combined with residual connections is used to avoid information loss caused by dimensionality compression. A modality-specific parameter matrix is ​​introduced during feature mapping to allow different modalities to retain unique feature distributions in a shared space. Simultaneously, cross-modal contrastive loss (such as InfoNCE) constrains the consistency of the mapped feature distribution, ensuring that the aligned feature vectors have semantic similarity in Euclidean space.

[0044] As a possible implementation of S130, when designing a cross-modal attention mechanism, a hybrid granularity attention model is constructed, which integrates global contextual attention and local region attention. This hybrid granularity attention model includes a global attention layer, a local attention layer, and hierarchical attention fusion.

[0045] In the global attention layer, standard multi-head self-attention (MSA) is employed to calculate the global correlation of feature sequences from different modalities, generating cross-modal context vectors. A modal gate mechanism is specifically designed to dynamically adjust the weights of each modality in the attention calculation, preventing any one modality from dominating information interaction. In the local attention layer, a position-sensitive attention mechanism is introduced to address the spatial locality of visual features (e.g., image patches, video frames) and the temporal locality of speech features (e.g., acoustic frames). Relative position encoding is added when calculating attention weights to enhance the preservation of intramodal structural information. Taking visual-language interaction as an example, when calculating attention between each image patch of visual features and each token of speech features, the spatial coordinates of the image patch and the syntactic position of the token are considered simultaneously. The hierarchical attention fusion representation constructs a three-level attention structure (intramodal self-attention → cross-modal mutual attention → global fusion attention), maintaining gradient flow through residual connections and layer normalization to avoid information decay in deep networks.

[0046] In some embodiments of S140, second-order correlations between modalities are calculated using bilinear interaction tensors; wherein the modalities include vision, language, and sound; based on the second-order correlations between modalities and aligned feature vectors, the aligned feature vectors are fused; based on the Transformer decoder architecture, the fused feature vectors are used as input to generate a unified multimodal representation through an autoregressive approach.

[0047] As a possible implementation of S140, it can be based on a dynamic routing architecture fusion module, which includes a Modal Interaction Unit (MIU) and a Semantic Integration Unit (SIU). The fusion module is used to fuse aligned feature vectors to generate a unified multimodal representation.

[0048] As one possible implementation of the modal interaction unit, the modal interaction unit calculates the second-order correlation between modes using a bilinear interaction tensor, with the formula: I = W v ·V·W l ·L T +W a ·A·W l ·L T +bias; where V represents the visual feature matrix, L represents the language feature matrix, A represents the vocal feature matrix, and W... v W is a visual modality-specific weight matrix. l W is a language modality-specific weight matrix. a The weight matrix is ​​specific to the vocal modality, and bias represents the bias term. Bias is a key parameter in neural networks used to adjust the flexibility of the model's output. Bias introduces additional degrees of freedom to the model, enabling it to learn more complex function mappings. Modal interaction units are used to capture nonlinear interactions between modalities, such as fine-grained alignment between visual images and language descriptions.

[0049] As a possible implementation of the semantic integration unit, a Transformer decoder architecture is adopted, using the fused interaction features as input to generate a unified multimodal representation through an autoregressive approach. A contrastive learning objective function is introduced, requiring different modal inputs of the same semantics to generate similar fused representations, while inputs of different semantics maintain sufficient differences to improve the discriminative power of the representation space.

[0050] In the feature alignment and fusion steps, taking the improved ParGo model as an example, a modal association token is added to the original local-global token design. This token dynamically encodes the correspondence between visual regions and language phrases and sound segments through learnable parameters. The attention masking mechanism is upgraded to a modal-aware mask, which preserves local region connectivity when calculating intra-modal attention and removes inter-modal masking restrictions when calculating cross-modal attention, achieving a three-level feature fusion of local detail preservation, global semantic alignment, and deep modal interaction.

[0051] In some embodiments, the large-model cross-modal collaborative understanding method may further include: converting a unified multimodal representation into a task-preset output based on a task-adaptive processing framework; wherein the task-adaptive processing framework includes a classification task, a generation task, and a retrieval task; during classification task processing, a multilayer perceptron classification head is accessed, and the feature weights of vision, language, and sound in the classification decision are configured based on preset weight parameters; during generation task processing, the fused unified multimodal representation is used as a context vector, and modality type information is fused through position encoding; during retrieval task processing, a query modality encoder and a database modality encoder are constructed, and the semantic distance between modalities is optimized by comparing loss functions.

[0052] In other words, downstream task processing can be configured to build a task-adaptive processing framework, supporting modular extensions for three main task categories: classification, generation, and retrieval. This enables efficient conversion from general multimodal representations to task-specific outputs. Therefore, it provides excellent scalability for large multimodal models, supporting various downstream tasks such as text and image retrieval, speech recognition, and video understanding, meeting the needs of different application scenarios.

[0053] As a possible implementation for classification tasks, this approach, targeting tasks such as image classification and sentiment classification, incorporates a multilayer perceptron classification head after the fusion representation layer. This head includes an adaptive dropout layer (dynamically adjusting the dropout rate based on the input modality complexity) and label smoothing regularization. For multi-label classification, a sigmoid activation function is used instead of softmax, combined with FocalLoss to address class imbalance. A specially designed modality weight adjustment module allows users to configure the importance of visual / language / sound features in classification decisions via hyperparameters. For example, in trademark classification, visual feature weights can be increased, while in speech sentiment classification, sound feature inputs can be strengthened.

[0054] As a possible implementation of generative tasks, tasks such as image caption generation and speech synthesis employ an encoder-decoder architecture. The encoder uses the fused multimodal representation as a context vector, fusing modal type information (e.g., adding visual, linguistic, and vocal identifier embeddings) through positional encoding. The decoder can be built based on the Transformer decoder, introducing a cross-attention mechanism to align the generated sequence with the multimodal representation, supporting autoregressive generation and beam search. In video caption generation, a temporal gate unit is added to dynamically integrate frame-level features and global video features. For text-to-speech (TTS) tasks, a spectrogram prediction module is designed, combining Mel-spectrum prediction with waveform generation networks (such as WaveNet) to achieve highly natural speech generation.

[0055] As a possible implementation of retrieval tasks, cross-modal retrieval (such as image-text retrieval and voice-image retrieval) can adopt a dual-tower model architecture, constructing a query modality encoder and a database modality encoder respectively. Semantic distance between modalities is optimized using contrastive loss functions (such as Triplet Loss and N-Pair Loss). In image-text retrieval, Region-Word Alignment Loss is introduced to force semantic matching between image region features and corresponding descriptive words. The retrieval system supports two modes: vector retrieval based on exact matching (using libraries such as FAISS for efficient nearest neighbor search) and cross-modal matching based on semantic reasoning. Cross-modal similarity scores are calculated using cosine similarity or dot product, and threshold filtering or learning to rank are combined to improve retrieval accuracy.

[0056] Finally, for large multimodal models with cross-modal collaboration, a multi-stage training strategy can be designed, including three stages: pre-training initialization, task adaptation training, and cross-modal alignment optimization, combined with dynamic loss balancing techniques to improve the model's generalization ability.

[0057] As a possible implementation of model training and optimization, a multi-task learning framework can construct a three-level task system including basic modal tasks, cross-modal alignment tasks, and downstream target tasks. The basic modal tasks involve single-modal pre-training for vision (image classification), language (masked language model), and sound (speech recognition), initializing feature extractors for each modality. Next, in the cross-modal alignment tasks, three core pre-training tasks are designed: modal translation tasks (image-to-text description generation, speech-to-text transcription), forcing semantic alignment between modalities through generation tasks; contrast matching tasks (determining whether image-text pairs and speech-image pairs semantically match), trained using binary cross-entropy loss (BCE Loss); and cross-modal masking tasks (randomly masking some features of a modality, requiring the model to recover them using information from other modalities, such as reconstructing using language and sound features after masking visual features). Finally, the downstream target tasks add task-specific loss functions based on specific application scenarios (such as medical image diagnosis, intelligent customer service), supporting end-to-end joint training.

[0058] As a possible approach to model training and optimization, loss function combination optimization can also be performed, employing a dynamic weight adjustment strategy to calculate the importance of each loss function in real time.

[0059]

[0060] In the above formula, Indicates the loss of the current batch, μ t Let λ represent the historical average loss, k represent the adjustment parameter, and λ represent the λ value.t This indicates the importance of the loss function.

[0061] The core losses include:

[0062] Cross-entropy loss (classification task): In the formula, This represents the cross-entropy loss value, used to measure the difference between the probability distribution predicted by the model and the true distribution. The smaller the value, the closer the model's predictions are to the actual results, and the better the model's performance. i To represent the actual label, one-hot encoding is typically used. This represents the probability value predicted by the model, and represents the probability that the sample belongs to the i-th class.

[0063] Loss generation (Seq2Seq task): In the formula, The generation loss value, w, is used to measure the quality of the sequence generated by the model. t w represents the word at position t in the target sequence. <t p(w) represents all words preceding the t-th position in the target sequence. context represents contextual information, typically generated by the encoder, and contains a global representation of the input sequence. t |w <t (context) represents the model's prediction of the word at position t as w, given the context and preceding words. t The probability of;

[0064] Contrast loss (cross-modal retrieval): In the formula, The contrastive loss value is used to measure the similarity of sample pairs within a common subspace. Its purpose is to minimize the distance between similar sample pairs and maximize the distance between dissimilar sample pairs. s(x,y) represents the similarity between sample x and sample y, and T represents the temperature coefficient, used to control the sharpness of the similarity distribution. The larger the T value, the smoother the similarity distribution; the smaller the T value, the sharper the similarity distribution. and This means that the similarity is converted into a probability distribution form using an exponential function;

[0065] Alignment loss (feature fusion): In the formula, The alignment loss value is used to measure the degree of alignment between different modal features in the projection space. Its purpose is to map features of different modalities into a unified space, ensuring that they can be compared and combined. proj(v), proj(l), and proj(a) represent the representations of different modal features such as visual, text, and audio in the projection space, respectively. The projection operation is usually implemented through linear or nonlinear transformations.

[0066] In the optimization of loss function combination, taking the improvement of VALOR model as an example, a hierarchical grouping strategy is introduced in the MGA (Multimodal Grouping Alignment) task to divide visual regions, language phrases, and sound segments into semantic groups of different granularities (word level, phrase level, paragraph level). Cross-modal structural alignment is enhanced through inter-group contrast loss. The MGC (Multimodal Grouping Description) task is upgraded to a conditional generation task, requiring the model to generate a detailed description of the target modality based on the grouping features of other modalities, thereby strengthening the semantic generation capability between modalities.

[0067] As a possible implementation of model training and optimization, optimization strategies and training techniques can also include: employing mixed-precision training (FP16 / FP32) combined with gradient accumulation techniques to support training ultra-large-scale models with limited GPU memory. Regarding optimizer selection, AdamW (weight decay + gradient clipping) is used in the pre-training phase, switching to RAdam or AdaFactor in the fine-tuning phase to adapt to task-specific parameter updates. Cosine annealing is used for learning rate scheduling, combined with a warm-up phase to avoid gradient explosion in the early stages of training. Model regularization includes weight decay, dropout, and adversarial training (FGSM / FGSM) to improve the model's robustness to noisy inputs. In distributed training, a strategy combining data parallelism and model parallelism is adopted to support training large cross-modal models with hundreds of billions of parameters.

[0068] Figure 2 This is a schematic diagram of the structure of a large-model cross-modal collaborative understanding method device provided in an exemplary embodiment of the present invention, as shown below. Figure 2As shown, the device 2 for the large-scale cross-modal collaborative understanding method includes: a preprocessing module 21, which preprocesses visual data, language data, and sound data to obtain preprocessed visual data, language data, and sound data; wherein, the preprocessing includes: adaptive size adjustment and normalization of visual data, word segmentation and dynamic truncation of language data, and bandpass filtering and spectral subtraction noise reduction of sound data; a feature extraction module 22, which extracts visual features, language features, and sound features based on the preprocessed visual data, language data, and sound data; an alignment module 23, which aligns visual features, language features, and sound features based on an adaptive mapping network and a hybrid granularity cross-modal attention mechanism to obtain aligned feature vectors; and a fusion module 24, which fuses the aligned feature vectors based on a dynamic routing architecture to generate a unified multimodal representation.

[0069] In one embodiment, the visual data includes image data and video data. The preprocessing module 21 can be configured to use an adaptive size adjustment algorithm for image data to unify the resolution to a preset size while maintaining the key features of the image data; for video data, key frame extraction is performed, and a motion detection algorithm based on optical flow is used to filter frames with dynamic changes greater than a preset change value, converting the continuous video stream into a preset number of frame sequences.

[0070] In one embodiment, the language data includes Chinese data and English data. The preprocessing module 21 can be configured to perform word segmentation on the Chinese data and English data. The Chinese data is segmented using the jieba tool library, and the English data is segmented using the NLTK tool library. When the length of the language data exceeds a preset length, key semantic segments are retained through dynamic truncation based on a sliding window truncation strategy combined with a preset maximum input length limit. A spelling correction module based on the perplexity of the language model performs spelling correction on the segmented Chinese data and English data.

[0071] In one embodiment, the preprocessing module 21 can be configured to perform a multi-stage noise reduction process on the sound data; wherein, the multi-stage noise reduction process includes removing the environmental noise frequency band through a bandpass filter, separating the speech segment and the silence segment using a speech activity detection technology based on a hidden Markov model, and using spectral subtraction or a deep learning noise reduction model to denoise the speech segment; wherein, preprocessing the visual data, language data and sound data to obtain the preprocessed visual data, language data and sound data further includes: converting sound data from different sources into a target sampling rate; performing endpoint detection and prosodic feature extraction on the sound data, and generating a Mel spectrogram or linearly predicted cepstral coefficients.

[0072] In one embodiment, the feature extraction module 22 can be configured to: construct a three-level feature extraction system to extract visual data, language data and sound data into visual features, language features and sound features of abstract semantics; wherein, the three-level feature extraction system includes: basic feature extraction, task adaptation fine-tuning and cross-modal enhancement preprocessing.

[0073] In one embodiment, the alignment module 23 can be configured to: based on an adaptive mapping network, perform dimensional matching on visual features, language features, and sound features of different dimensions through basic dimensional alignment, semantic space calibration, and cross-modal interaction enhancement to obtain feature vectors, and introduce a modality-specific parameter matrix to ensure that different modalities retain unique feature distributions in a shared space; constrain the consistency of the mapped feature distributions through cross-modal comparative loss to ensure that the semantic similarity of the aligned feature vectors is greater than a preset similarity threshold; and construct a hybrid granularity attention model that integrates global contextual attention and local region attention.

[0074] In one embodiment, the alignment module 23 can also be configured to: calculate the global correlation of feature sequences of different modalities and generate cross-modal context vectors; wherein the modalities include vision, language and sound; and construct a position-sensitive attention mechanism for the spatial locality of visual features and the temporal locality of language features and sound features, and add relative position encoding when calculating attention weights in order to preserve the structural information within the modality.

[0075] In one embodiment, the fusion module 24 can be configured to: calculate the second-order correlation between modalities using a bilinear interaction tensor; wherein the modalities include vision, language, and sound; fuse the aligned feature vectors based on the second-order correlation between modalities and the aligned feature vectors; and generate a unified multimodal representation by using the fused feature vectors as input based on the Transformer decoder architecture and an autoregressive approach.

[0076] In one embodiment, the large-model cross-modal collaborative understanding method device 2 may further include: converting a unified multimodal representation into a task-preset output based on a task-adaptive processing framework; wherein the task-adaptive processing framework includes a classification task, a generation task, and a retrieval task; during the classification task processing, a multilayer perceptron classification head is accessed, and the feature weights of vision, language, and sound in the classification decision are configured based on preset weight parameters; during the generation task processing, the fused unified multimodal representation is used as a context vector, and modality type information is fused through position encoding; during the retrieval task processing, a query modality encoder and a database modality encoder are constructed, and the semantic distance between modalities is optimized by comparing loss functions.

[0077] This invention provides a device for a large-scale model cross-modal collaborative understanding method. The device can be implemented in software, hardware, or a combination of both. From a hardware perspective, in addition to a CPU, memory, network interface, and non-volatile memory, the device in the embodiment typically includes other hardware, such as a forwarding chip responsible for processing packets. Taking software implementation as an example, as a logical device, it is formed by the CPU of the device loading the corresponding computer program instructions from the non-volatile memory into memory for execution.

[0078] According to another aspect of the present invention, a computer-readable storage medium is provided, the storage medium storing a computer program for performing the large model cross-modal collaborative understanding method of any of the above embodiments.

[0079] In addition to the methods and devices described above, embodiments of the present invention may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the large model cross-modal collaborative understanding methods described above according to various embodiments of the present invention.

[0080] According to another aspect of the present invention, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; and a processor for executing the large-model cross-modal collaborative understanding method of any of the above embodiments.

[0081] Furthermore, embodiments of the present invention may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps in the large model cross-modal collaborative understanding method described above according to various embodiments of the present invention.

[0082] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for cross-modal collaborative understanding of large models, characterized in that, include: Visual data, language data, and sound data are preprocessed to obtain preprocessed visual data, language data, and sound data; wherein, the preprocessing includes: adaptive size adjustment and normalization of the visual data, word segmentation and dynamic truncation of the language data, and bandpass filtering and spectral subtraction noise reduction of the sound data; Based on the preprocessed visual data, language data, and sound data, visual features, language features, and sound features are extracted. Based on an adaptive mapping network and a hybrid granularity cross-modal attention mechanism, the visual features, the language features, and the sound features are aligned to obtain aligned feature vectors. Based on a dynamic routing architecture, aligned feature vectors are fused to generate a unified multimodal representation.

2. The large-scale model cross-modal collaborative understanding method according to claim 1, characterized in that, The visual data includes image data and video data. Adaptive size adjustment and normalization are performed on the visual data, including: For image data, an adaptive size adjustment algorithm is used to unify the resolution to a preset size while maintaining the key features of the image data; For keyframe extraction of video data, a motion detection algorithm based on optical flow is used to filter frames with dynamic changes greater than a preset change value, and the continuous video stream is converted into a preset number of frame sequences.

3. The large-scale model cross-modal collaborative understanding method according to claim 1, characterized in that, The language data includes Chinese and English data. The language data undergoes word segmentation and dynamic truncation, including: The Chinese data and the English data are segmented into words; the Chinese data is segmented using the jieba tool library, and the English data is segmented using the NLTK tool library. When the length of the language data exceeds the preset length, key semantic segments are retained through dynamic truncation based on the sliding window truncation strategy and in combination with the preset maximum input length limit. The spelling correction module based on language model perplexity performs spelling correction on the segmented Chinese and English data.

4. The large-scale model cross-modal collaborative understanding method according to claim 1, characterized in that, Bandpass filtering and spectral subtraction are performed on the audio data, including: The audio data undergoes a multi-stage noise reduction process; wherein, the multi-stage noise reduction process includes removing the ambient noise frequency band through a bandpass filter, separating the speech segment from the silence segment using a speech activity detection technique based on a hidden Markov model, and using spectral subtraction or a deep learning noise reduction model to reduce the noise of the speech segment; The process of preprocessing visual data, language data, and sound data to obtain preprocessed visual data, language data, and sound data further includes: Convert the audio data from different sources to the target sampling rate; Endpoint detection and prosodic feature extraction are performed on the sound data to generate Mel spectrograms or linearly predicted cepstral coefficients.

5. The large-scale model cross-modal collaborative understanding method according to claim 1, characterized in that, Based on the preprocessed visual data, language data, and sound data, visual features, language features, and sound features are extracted, including: A three-level feature extraction system is constructed to extract the visual data, language data, and sound data into abstract semantic visual features, language features, and sound features; wherein, the three-level feature extraction system includes: basic feature extraction, task adaptation fine-tuning, and cross-modal enhancement preprocessing.

6. The large-scale model cross-modal collaborative understanding method according to claim 1, characterized in that, Based on an adaptive mapping network and a hybrid granularity cross-modal attention mechanism, the visual features, the language features, and the sound features are aligned to obtain aligned feature vectors, including: Based on an adaptive mapping network, the visual features, language features, and sound features of different dimensions are matched by basic dimension alignment, semantic space calibration, and cross-modal interaction enhancement to obtain feature vectors. Furthermore, a modality-specific parameter matrix is ​​introduced to ensure that different modalities retain unique feature distributions in a shared space. By constraining the consistency of feature distributions after cross-modal comparative loss, the semantic similarity of aligned feature vectors is made greater than a preset similarity threshold. Construct a hybrid granularity attention model that integrates global contextual attention and local region attention.

7. The large-scale model cross-modal collaborative understanding method according to claim 6, characterized in that, Construct a hybrid granularity attention model that integrates global contextual attention and local region attention, including: Calculate the global correlation of feature sequences from different modalities to generate cross-modal context vectors; wherein the modalities include visual, language, and sound. To address the spatial locality of the visual features and the temporal locality of the language and sound features, a position-sensitive attention mechanism is constructed. Relative position encoding is incorporated when calculating attention weights to preserve intramodal structural information.

8. The large-scale model cross-modal collaborative understanding method according to claim 1, characterized in that, Based on a dynamic routing architecture, aligned feature vectors are fused to generate a unified multimodal representation, including: The second-order correlation between modalities is calculated using a bilinear interaction tensor; wherein the modalities include visual, language, and sound. Based on the second-order correlation between modes and the aligned feature vectors, the aligned feature vectors are fused. Based on the Transformer decoder architecture, the fused feature vectors are used as input to generate a unified multimodal representation through an autoregressive approach.

9. The large-scale model cross-modal collaborative understanding method according to claim 1, characterized in that, Large-scale model cross-modal collaborative understanding methods also include: Based on the task adaptive processing framework, the unified multimodal representation is converted into a task preset output; wherein, the task adaptive processing framework includes classification tasks, generation tasks, and retrieval tasks; When processing the classification task, a multilayer perceptron classification head is connected, and the feature weights of vision, language and sound in the classification decision are configured based on the preset weight parameters. During the generation task processing, the fused unified multimodal representation is used as the context vector, and the modality type information is fused through position encoding. During the retrieval task processing, a query modality encoder and a database modality encoder are constructed, and the semantic distance between modalities is optimized by comparing loss functions.

10. A method and apparatus for cross-modal collaborative understanding of large models, characterized in that, include: The preprocessing module preprocesses the visual data, language data, and sound data to obtain the preprocessed visual data, language data, and sound data. The preprocessing includes: adaptive size adjustment and normalization of the visual data, word segmentation and dynamic truncation of the language data, and bandpass filtering and spectral subtraction noise reduction of the sound data. The feature extraction module extracts visual features, language features, and sound features based on the preprocessed visual data, language data, and sound data. The alignment module, based on an adaptive mapping network and a hybrid granularity cross-modal attention mechanism, aligns the visual features, the language features, and the sound features to obtain an aligned feature vector. The fusion module, based on a dynamic routing architecture, fuses aligned feature vectors to generate a unified multimodal representation.

Images