Unified representation method for multi-modal heterogeneous features based on large model semantic coding

Abstract

The application discloses a multi-modal heterogeneous feature unified representation method based on large model semantic coding, relates to the technical field of multi-modal data processing, and comprises the following steps: S1, collecting original data of at least two different modes, performing exclusive preprocessing on the mode data, extracting initial features of the modes, and performing standardization processing to obtain a standardized heterogeneous initial feature set; the application realizes deep adaptation of heterogeneous features and a large model semantic space by constructing a mode exclusive adaptation module combined with a fine-grained feature semantic mask protection mechanism, completely retains fine-grained features of non-text modes, avoids semantic loss in the mode conversion process, realizes deep semantic modeling of heterogeneous features by deep semantic coding of a large model combined with inter-mode causal correlation mining, avoids false semantic alignment, realizes accurate alignment of cross-modal semantics by a double semantic alignment and dynamic semantic space anchoring mechanism, and solves the long-standing mode gap problem.

Description

Technical Field

[0001] This invention relates to the field of multimodal data processing technology, specifically to a unified representation method for multimodal heterogeneous features based on large model semantic coding. Background Technology

[0002] With the rapid iteration of artificial intelligence technology, the perception and decision-making capabilities of intelligent systems have gradually evolved from the single-modal processing stage to the multimodal collaborative intelligence stage. Multi-source heterogeneous data such as text, images, audio, and sensor signals have become the core input carriers of current intelligent systems. The ability to fuse and collaboratively process multimodal data directly determines the performance ceiling of intelligent systems in complex real-world scenarios and is also a core research direction for the practical application of artificial intelligence. However, the underlying features of different modalities exhibit inherent heterogeneity: text features are centered on sequential semantic logic, image features on spatial structure and texture distribution, and audio and sensor signals on the physical change patterns of time. These heterogeneous features have natural barriers in dimensional distribution, semantic expression, and information encoding logic, becoming the core bottleneck restricting the efficient collaborative utilization of multimodal data.

[0003] To address these issues, the industry has successively introduced multiple generations of heterogeneous feature processing technologies. Early traditional feature fusion methods were mainly divided into three categories: early fusion (concatenating raw data layers), mid-term fusion (concatenating and weighting feature layers), and late fusion (integrating decision-level results). These methods only performed simple linear integration along the data or feature dimensions, without introducing deep semantic modeling mechanisms. They failed to solve the semantic misalignment problem of heterogeneous features, and their performance deteriorated sharply in scenarios with significant modal differences, exhibiting extremely poor generalization ability. With the development of deep learning technology, cross-modal representation methods based on autoencoders, generative adversarial networks, and contrastive learning have gradually become mainstream. These methods construct dedicated cross-modal mapping networks to map features from different modalities to the same high-dimensional space, alleviating the modality gap problem to some extent. However, the generalization ability of such mapping networks is limited, the adaptability to niche modalities and scarce labeled data is poor, and they do not utilize the general semantic modeling ability of large models. The semantic level of feature representation is shallow and cannot capture complex cross-modal semantic relationships. At the same time, when the number of modalities increases, the model complexity increases exponentially, the resource overhead of training and inference increases sharply, and it is difficult to meet the needs of real-world scenarios.

[0004] In recent years, large language models and multimodal large models have made breakthroughs in single-modal intelligence tasks due to their powerful general semantic modeling capabilities. The industry has also begun to explore introducing large models into the field of unified representation of heterogeneous features. Existing solutions mainly fall into two categories: one is to convert non-textual modal features into feature vectors using a dedicated encoder and then simply concatenate them with the text semantic vectors output by the large model; the other is to convert non-textual modal features into text descriptions through cue word engineering and then input them into the large model for semantic encoding. However, these solutions only use the large model as a post-processing tool and fail to achieve deep integration of heterogeneous features with the semantic space of the large model. During modality conversion, the fine-grained features of non-textual modalities are severely lost, and semantic alignment remains only at the surface level, failing to achieve deep semantic consistency. Furthermore, these solutions rely on external model interfaces, resulting in insufficient adaptability and stability, making it difficult to support high-performance operation in complex scenarios.

[0005] In summary, existing heterogeneous feature unified representation techniques cannot simultaneously achieve deep adaptation between heterogeneous features and the semantic space of large models, accurate alignment of cross-modal semantics, and complete preservation of fine-grained features of non-textual modalities. They also suffer from weak model generalization ability, poor adaptability to new modalities and scarce data, and high training and inference costs, making it difficult to support the high-performance and high-reliability operation of multimodal intelligent systems in complex scenarios. Therefore, it is of great significance to develop a multimodal heterogeneous feature unified representation method based on large model semantic encoding. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a unified representation method for multimodal heterogeneous features based on large-model semantic encoding. This method breaks down the modal barriers of heterogeneous features, achieving deep adaptation and precise semantic alignment of different modal features within the semantic space of a large model, thus solving the long-standing problems of modality gap and semantic misalignment. It also achieves complete preservation of fine-grained features of non-textual modalities, avoiding semantic loss during modality conversion and encoding, and improving the accuracy and completeness of representation results. Furthermore, it enhances the model's generalization and adaptation capabilities to new modalities, niche modalities, and scarce data, reducing resource overhead for model training and inference, and avoiding the complexity explosion problem caused by an increase in the number of modalities. Finally, it constructs a unified representation scheme that can be implemented throughout the entire process, directly adapting to various multimodal downstream tasks and improving the overall performance and scenario adaptability of multimodal intelligent systems.

[0007] To address the aforementioned technical problems, this invention provides the following technical solution: a unified representation method for multimodal heterogeneous features based on large-model semantic encoding, comprising the following steps: S1. Collect raw data from at least two different modalities, perform dedicated preprocessing for each modal data, extract the initial features of each modality and perform standardization processing to obtain a standardized heterogeneous initial feature set; S2. For the initial features of different modalities, construct mutually independent modality-specific adaptation subnetworks, map the initial features of each modality to the dimension that matches the semantic space of the large model, perform semantic enhancement processing on the fine-grained features of each modality, and obtain the semantically enhanced intermediate features of each modality. S3. Input the semantic enhancement intermediate features of each modality into the semantic coding layer of the pre-trained large model to perform deep semantic coding, introduce residual connections and normalization processing to optimize the coding results, and obtain the semantic coding features of each modality. S4. Based on the semantic encoding features of the text modality, a dual semantic alignment mechanism combining cross-modal attention alignment and contrastive learning alignment is constructed. Semantic alignment processing is performed on the semantic encoding features of each modality to obtain an aligned heterogeneous feature set. S5. Dynamically allocate fusion weights based on the semantic contribution of each modality alignment feature, perform weighted fusion, perform adaptive filtering on the fused features, and output a unified representation result of multimodal heterogeneous features with fixed dimensions. S6. Based on the performance feedback from downstream tasks, iteratively optimize the parameters of the modality-specific adaptor subnetwork, the large model encoding parameters, the alignment mechanism loss weights and fusion weights, and update the unified representation results.

[0008] Furthermore, step S1 includes the following steps when performing dedicated preprocessing for each modality of data: For the raw text modal data, text cleaning, word segmentation, stop word removal and entity resolution are performed, and the text data is converted into initial text features using a word embedding model; For the raw image modal data, size normalization, pixel value normalization and data augmentation processing are performed, and the initial visual features of the image are extracted using a visual encoder; For the original audio modal data, resampling, denoising and short-time Fourier transform processing are performed to extract the Mel frequency cepstral coefficients, time domain features and frequency domain features of the audio, and convert them into the initial audio features; For the raw sensor signal modal data, filtering, noise reduction, and normalization are performed to convert the physical signals into initial sensor features, and spatiotemporal alignment processing of multimodal data is performed.

[0009] Furthermore, the standardization process performed in step S1 adopts the L2 normalization method, which performs sample-by-sample norm normalization on the initial features extracted from each modality, unifying the norm of all initial feature vectors to a fixed value. The spatiotemporal alignment of multimodal data adopts a combination of hardware timestamp synchronization and software time compensation to unify the time base of different modal data. The initial features of all modalities are stored in a two-dimensional tensor format. The first dimension of the tensor is the feature sequence length, and the second dimension of the tensor is the feature dimension of the corresponding modality. The high-resolution image uses sliding window segmentation and feature stitching to extract the initial visual features.

[0010] Furthermore, in step S2, each modality-specific adaptor subnetwork constructed adopts a combination structure of fully connected layers, normalization layers, and activation layers. The training parameters of each adaptor subnetwork are independent of each other, the input of each adaptor subnetwork is the standardized initial features of the corresponding modality, and the output dimension of each adaptor subnetwork is consistent with the input dimension of the semantic encoding layer of the pre-trained large model. The semantic enhancement intermediate features of the text modality are generated through a weighted fusion method, and the fusion formula is: , in the formula To enhance the semantic meaning of the text, intermediate features are used, where α is the weight coefficient of the intermediate features. The value of α is determined based on the statistical results of the semantic contribution of the text modal features. To adapt intermediate features for text output, Semantic vectors are embedded into pre-trained words for the large model. The semantically enhanced intermediate features from non-textual modalities are generated through weighted fusion guided by cue words. The fusion formula is as follows: , in the formula For non-textual modal semantic enhancement intermediate features, β is the weight coefficient of the non-textual intermediate features, and the value of β is determined based on the requirements for preserving fine-grained features of the non-textual modality. For non-text adaptation, output intermediate features. The semantic vector generated to guide the prompt words.

[0011] Furthermore, the text modality weight coefficient α is determined through multiple sets of text feature semantic expression integrity tests. During the test, text data of different lengths and types are selected, and the semantic matching degree of the semantic enhancement intermediate features under different coefficients is statistically analyzed. The value with the highest semantic matching degree is selected as the final coefficient. The non-text modality weight coefficient β is determined through fine-grained feature retention rate tests. During the test, fine-grained features related to image texture, audio emotion, and sensor signal changes are collected, and the retention degree of fine-grained features under different coefficients is statistically analyzed. The value with the best retention degree is selected as the final coefficient.

[0012] Furthermore, step S3 includes the following steps when performing deep semantic encoding: Configure the encoding parameters of the pre-trained large model, set the context window length, number of encoding layers, number of attention heads and dropout probability, and fix the parameters of the backbone network of the pre-trained large model; For semantic enhancement intermediate features of text modalities, they are directly input into the semantic encoding layer of the pre-trained large model to perform deep semantic encoding processing; For semantic enhancement intermediate features of non-text modalities, they are concatenated with the semantic prompt text of the corresponding modalities, and then input into the semantic encoding layer of the pre-trained large model to perform deep semantic encoding processing; LayerNorm normalization is used to process the encoded output results, and residual connections are introduced to fuse the semantically enhanced intermediate features with the encoded output features to obtain the semantic encoding features of each modality.

[0013] Furthermore, in step S4, the text modality semantic encoding features used are cross-modal alignment baseline features. Cross-modal attention alignment employs a multi-head attention mechanism, using non-text modality semantic encoding features as query terms and text modality semantic encoding features as key and value terms. The semantic correlation between each modality feature and the baseline feature is calculated. Contrastive learning alignment uses the InfoNCE loss function to optimize the feature space distribution. The loss function formula is: , in the formula To compare the learning loss values, N is the number of positive sample pairs and M is the number of negative sample pairs. This is the function for calculating feature similarity. For the current modality encoding features, For features corresponding to the same semantic modality, Encoding features for negative samples The temperature coefficient is determined based on the requirement of smoothing the distribution in the characteristic space.

[0014] Furthermore, the positive sample pairs in the contrastive learning loss function are combinations of encoded features from different modalities within the same semantic scene, while the negative sample pairs are combinations of encoded features from arbitrary modalities within different semantic scenes. (Temperature coefficient) The temperature coefficient is used to control the distribution range of similarity values. The value is determined by the clustering effect test in the feature space. During the test, the temperature coefficient value is adjusted, the cluster compactness of the same semantic features and the degree of separation of different semantic features are observed, and the value with the best clustering effect is selected as the final temperature coefficient.

[0015] Furthermore, step S5 includes the following steps when performing weighted fusion and adaptive filtering: Cosine similarity is used to calculate the semantic similarity between each modality alignment feature and the text modality baseline feature. The similarity calculation formula is as follows: , in the formula Let A be the feature cosine similarity, A be the modality alignment feature to be calculated, and B be the text modality baseline alignment feature. and These are the L2 norms of the corresponding features; Based on the cosine similarity results, the fusion weights for each modality are generated by normalization, and the sum of all weight values ​​is 1; Weighted fusion is performed according to the fusion weights, and the fusion formula is as follows: , in the formula The features are those after fusion, where n is the total number of modes. Let i be the fusion weights for the i-th mode. The alignment feature is for the i-th mode; A feature selection network consisting of a fully connected layer and a Sigmoid activation layer is constructed. Feature dimensions are selected according to a preset threshold, and a unified representation result of multimodal heterogeneous features with fixed dimensions is output.

[0016] Furthermore, in step S2, the image modality adaptor network introduces an attention mechanism to strengthen the fine-grained feature weights, the audio modality adaptor network performs feature mapping in conjunction with emotional features, and the sensor signal modality adaptor network injects physical constraints to complete feature transformation. All non-text modality adaptor networks convert their output features into discrete feature units that match the semantic space of the large model. The dimension of the discrete feature units is consistent with the dimension of the output features of the text modality adaptor network. During the semantic enhancement process, the fine-grained feature dimensions related to texture, emotion, and physical changes of the non-text modality are preserved, and the numerical distribution range of the fine-grained features is not compressed.

[0017] Compared with existing technologies, this unified representation method for multimodal heterogeneous features based on large-model semantic encoding has the following advantages: This invention achieves deep adaptation of heterogeneous features to the semantic space of a large model by constructing a modality-specific adaptation module combined with a fine-grained feature semantic mask protection mechanism. It fully preserves the fine-grained features of non-textual modalities, avoiding semantic loss during modality conversion. Through deep semantic encoding of the large model combined with causal correlation mining between modalities, it achieves deep semantic modeling of heterogeneous features, avoiding false semantic alignment. Through dual semantic alignment and dynamic semantic space anchoring mechanism, it achieves accurate alignment of cross-modal semantics, solving the long-standing modality gap problem. Through incremental modality adaptation strategy, it significantly improves the model's generalization ability to new modalities and scarce data, while reducing the resource overhead of training and inference. It can directly adapt to various multimodal downstream tasks, improving the overall performance of multimodal intelligent systems.

[0018] Other advantages, objectives and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination or study, or may be learned from the practice of the invention. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0020] Figure 1The flowchart shows a unified representation method for multimodal heterogeneous features based on large model semantic encoding. Figure 2 This is a flowchart illustrating the steps of a unified representation method for multimodal heterogeneous features based on large-model semantic encoding. Figure 3 This is a flowchart for performing dedicated preprocessing on data for each modality. Detailed Implementation

[0021] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0022] This invention provides a unified representation method for multimodal heterogeneous features based on large-model semantic encoding. Through a closed-loop design encompassing heterogeneous feature acquisition and standardized preprocessing, modality-specific adaptation and semantic enhancement, large-model deep semantic encoding, dual semantic alignment and feature fusion, unified representation output, and iterative optimization, it constructs independent modality-specific adaptation sub-networks to achieve deep adaptation between heterogeneous features and the large-model semantic space. Combined with a fine-grained feature semantic masking protection mechanism, it fully preserves core features such as texture, emotion, and physical changes in non-textual modalities. It relies on a pre-trained large model to complete deep semantic encoding and mine causal relationships between modalities. Using the text modality as a benchmark, it constructs a dual semantic alignment mechanism of cross-modal attention alignment and contrastive learning alignment. Through dynamic weighted fusion and adaptive feature selection, it outputs a unified representation result with fixed dimensions. This method can adapt to four heterogeneous modalities: text, image, audio, and sensor signals. It completely solves the industry's technical pain points of modality gap, semantic loss, poor generalization, and high resource overhead, providing high-performance feature support for various downstream tasks such as cross-modal retrieval, multimodal classification, intelligent reasoning, and human-computer interaction. The following detailed description is based on specific embodiments.

[0023] This embodiment selects cross-modal retrieval as a specific application scenario, and takes four heterogeneous modal data of text, image, audio, and tactile sensor signals as the processing objects. It uses publicly available multimodal datasets and measured sensor data to complete the method verification, fully reproduce the entire process of this invention from data processing to unified representation output and downstream task adaptation, and clearly demonstrate the execution logic parameter configuration and effect implementation method of each step.

[0024] This embodiment uses a server-level hardware environment for model training and inference. The CPU is a 24-core, 48-thread Intel Xeon Platinum 8375C processor, the GPU is an NVIDIA A100 graphics card with 80GB of VRAM, the RAM is 128GB, and the storage medium is a 2TB solid-state drive. The software environment uses Ubuntu 20.04 LTS operating system, Python 3.9 programming language, PyTorch 2.0 deep learning framework, an LLaMA-370B pre-trained large model for semantic encoding, a ViT-L / 14 visual encoder for image feature extraction, the Librosa library for audio processing, and the SciPy library for sensor signal processing. The experimental data consisted of text and image data from the MSCOCO dataset, audio data from the AudioCaps dataset, and 1,000 sets of tactile sensor pressure signal data for generalization testing. A total of 10,000 valid samples were selected from the dataset, with 8,000 sets used as the training set and 2,000 sets used as the test set. The samples covered various types of scenarios, including everyday scenes, animal scenes, and home scenes, to ensure the universality of the experimental results.

[0025] Heterogeneous feature acquisition and standardization preprocessing; such as Figure 3 As shown, this step completes the acquisition, modal preprocessing, and standardization of multimodal raw data, laying the foundation for subsequent feature adaptation and encoding.

[0026] First, heterogeneous feature acquisition was conducted, simultaneously collecting raw data from four modalities: text data (natural language descriptions), image data (color images of the corresponding scene), audio data (ambient sound and speech data of the corresponding scene), and sensor signal data (tactile pressure values ​​and torque vector data). After all data was collected, it was categorized and stored according to modal type, and a unified data index was established to ensure the orderly processing of subsequent data.

[0027] Subsequently, modality-specific preprocessing operations were performed. For text modality data, text cleaning, word segmentation, stop word removal, and entity resolution were completed sequentially. In the cleaning process, special characters, redundant spaces, and invalid information were removed. The word segmentation tool Jieba was used, the stop word removal tool Harbin Institute of Technology stop word list was used, and entity resolution was performed by replacing pronouns using the LLaMA-3 model. Finally, the BERT-base model was used to convert the text into 768-dimensional initial text features.

[0028] For image modal data, all images were uniformly scaled to 224×224 pixels, and pixel values ​​were normalized to the 0~1 range. Data augmentation was performed through random cropping and flipping. A ViT-L / 14 encoder was used to extract 1024-dimensional initial visual features. For high-resolution images, a sliding window segmentation and feature stitching strategy was employed to fully preserve texture details. For audio modal data, all audio data was resampled to 16000 Hz, and spectral gating was used for denoising. Short-time Fourier transform was used to extract the time-domain and frequency-domain features of the Mel-frequency cepstral coefficients, converting them into 512-dimensional initial audio features. For sensor signal modal data, a Battlewars filter was used for filtering and denoising. The physical signals were normalized and converted into 256-dimensional initial sensor features. A combination of hardware timestamp synchronization and software time compensation was used to achieve spatiotemporal alignment of the multimodal data, controlling the time alignment error to within 5ms.

[0029] Finally, feature standardization is performed. The L2 normalization method is used to normalize the norm of all initial features on a sample-by-sample basis, unifying the norm of all initial feature vectors to a fixed value, eliminating the calculation bias caused by the difference in feature dimensions of different modalities, and finally obtaining a standardized heterogeneous initial feature set, which includes four types of standardized initial features: text, image, audio, and sensor signals.

[0030] Modality-specific adaptation and semantic enhancement; such as Figure 1 As shown, this step constructs an independent modality-specific adaptive subnetwork to complete the mapping between heterogeneous features and the semantic space of the large model, and enhances the feature expression capability through semantic augmentation.

[0031] First, modality-specific adaptation sub-networks are constructed, with each of the four modalities corresponding to an independent adaptation sub-network. All sub-networks employ a combination of fully connected layers, normalization layers, and activation layers. The parameters of each sub-network are trained independently to avoid feature interference between modalities. The text modality adaptation sub-network takes 768-dimensional standardized text features as input, outputs 1024-dimensional features through two fully connected layers, and then sequentially connects to a BatchNorm normalization layer and a ReLU activation layer to obtain intermediate text features. The image modality adaptor network takes 1024-dimensional standardized image features as input, introduces the SE-Net attention mechanism to enhance fine-grained texture feature weights, and outputs 1024-dimensional features through one fully connected layer. These features are then connected to a BatchNorm normalization layer and a ReLU activation layer, simultaneously converting the image features into discrete feature units that match the semantic space of the large model, resulting in intermediate image features. The audio modality adaptor network takes 512-dimensional standardized audio features as input, and outputs 768-dimensional and 1024-dimensional features sequentially through three fully connected layers. These features are then connected to a BatchNorm normalization layer and a GELU activation layer, and combined with sentiment features extracted by an audio sentiment classifier, resulting in intermediate audio features. The sensor signal modality adaptor network takes 256-dimensional standardized sensor features as input, and outputs 512-dimensional and 1024-dimensional features sequentially through two fully connected layers. These features are then connected to a BatchNorm normalization layer and a ReLU activation layer, injecting physical constraints such as collision detection gravity constraints, resulting in intermediate sensor features. All intermediate feature dimensions output by the adaptor subnetworks are uniformly set to 1024, consistent with the semantic encoding input dimension of the large model.

[0032] Subsequently, semantic enhancement processing is performed. The text modality adopts a weighted fusion method using pre-trained semantic vectors. In the specific implementation of this embodiment, the calculation formula for the intermediate features of text semantic enhancement is as follows: ,in To enhance the semantic meaning of the text, intermediate features are used, where α is the weight coefficient of the intermediate text features. To adapt intermediate features for text output, Semantic vectors are embedded into the pre-trained words of the large model. The text modality weight coefficient α is determined through multiple sets of text feature semantic expression integrity tests. In the testing process, text data of different lengths and types are selected, and the semantic matching degree of the semantic enhancement intermediate features under different coefficients is statistically analyzed. Finally, the coefficient value of 0.7 with the highest semantic matching degree is selected as the coefficient value.

[0033] The non-text modality employs a cue-guided weighted fusion method. In the specific implementation of this embodiment, the calculation formula for the intermediate features of the non-text modality semantic enhancement is as follows: ,in For non-textual modal semantic enhancement intermediate features, β is the weight coefficient of the non-textual intermediate features. For non-text adaptation, output intermediate features. The semantic vector is generated to guide the prompt words. The non-textual modality weight coefficient β is determined through a fine-grained feature retention rate test. During the test, fine-grained features related to changes in image texture, audio emotion sensor signals are collected, and the degree of retention of fine-grained features under different coefficients is statistically analyzed. Finally, 0.6, which has the best retention rate, is selected as the coefficient value.

[0034] After semantic enhancement is completed, the semantic enhancement intermediate features of all modalities are integrated to form a unified set of intermediate features. All features have been mapped to the semantic space that the large model can recognize, and the fine-grained feature information of non-textual modalities is fully preserved.

[0035] Large-scale deep semantic encoding: This step relies on a pre-trained large-scale model to complete deep semantic encoding, explore the deep semantic relationships of heterogeneous features, and optimize the stability and expressive power of the encoded features.

[0036] First, the large model configuration and parameter settings were completed. The LLaMA-370B pre-trained large model was selected as the semantic encoding core. The parameters of the backbone network of the large model were fixed, and only the parameters related to the encoding layer were optimized. The context window length was configured to be 2048, the number of encoding layers was 32, the number of attention heads was 16, the dropout probability was set to 0.1, and the context memory compression strategy was adopted to reduce the memory usage of long text processing.

[0037] Subsequently, deep semantic encoding for different modalities is performed. The semantically enhanced intermediate features of the text modality are directly input into the semantic encoding layer of the large model. A self-attention mechanism is used to mine contextual semantic relationships, outputting text semantic encoded features. The semantically enhanced intermediate features of the non-text modality are concatenated with corresponding prompts and input into the semantic encoding layer. Prompts are generated based on modal features: image prompts describe scene and texture information, audio prompts describe sound type and emotional information, and sensor prompts describe physical quantity changes. The large model completes semantic parsing of the non-text modality guided by the prompts, outputting the corresponding semantic encoded features. The encoding process borrows from earlier fusion strategies, projecting non-text modal features and text prompts from the initial stage to a shared representation space, improving encoding consistency.

[0038] Finally, the encoding feature optimization is performed. The LayerNorm normalization method is used to process the encoding output to eliminate the gradient vanishing problem. At the same time, residual connections are introduced to directly fuse the semantically enhanced intermediate features with the semantically encoded features, retaining the original fine-grained feature information. Finally, the optimized semantically encoded features of the four modalities are obtained, and all feature dimensions are 1024.

[0039] Dual semantic alignment and feature fusion: This step constructs a dual semantic alignment mechanism to achieve accurate cross-modal semantic alignment, and outputs a unified representation result through dynamic weighted fusion and adaptive filtering.

[0040] First, a dual semantic alignment mechanism is constructed, using text semantic encoding features as the cross-modal alignment benchmark. The first layer is cross-modal attention alignment, employing an 8-head attention mechanism. Non-textual modal semantic encoding features are used as query terms, and textual modal semantic encoding features are used as keys and values. The semantic correlation between each modal feature and the benchmark feature is calculated, and the feature weights are dynamically adjusted to achieve initial semantic alignment. The second layer is contrastive learning alignment, using the InfoNCE loss function to optimize the feature space distribution. In the specific implementation of this embodiment, the formula for calculating the contrastive learning loss function is: ,in To compare the learning loss values, N is the number of positive sample pairs and M is the number of negative sample pairs. Feature similarity calculation function For the current modality encoding features, For features corresponding to the same semantic modality, Encoding features for negative samples This is the temperature coefficient.

[0041] Positive sample pairs are combinations of encoded features from different modalities within the same semantic scenario, while negative sample pairs are combinations of encoded features from arbitrary modalities within different semantic scenarios. (Temperature coefficient) The temperature coefficient was determined through clustering performance testing in the feature space. The coefficient values ​​were adjusted to observe the clustering compactness of semantic features and the separation degree between different semantic features. The optimal value of 0.5 was ultimately selected as the temperature coefficient. In this embodiment, the batch size for contrastive learning was set to 64, the number of positive sample pairs to 32, the number of negative sample pairs to 128, and the loss weight to 0.5. The loss function minimizes the distance between positive and negative features, forcing all modal features to achieve precise alignment in the semantic space.

[0042] After alignment is completed, alignment feature optimization is performed. The L2 normalization method is used to process all aligned features to eliminate feature numerical differences and obtain a standardized alignment feature set.

[0043] Subsequently, weighted fusion and adaptive feature selection are performed. First, cosine similarity is used to calculate the semantic similarity between each modality alignment feature and the text baseline feature. In the specific implementation of this embodiment, the cosine similarity calculation formula is as follows: ,in Let A be the feature cosine similarity, A be the modality alignment feature to be calculated, and B be the text modality baseline alignment feature. and These are the L2 norms of the corresponding features. Based on the cosine similarity results, the fusion weights for each modality are generated by normalization: text feature weight is 0.35, image feature weight is 0.3, audio feature weight is 0.2, sensor signal feature weight is 0.15, and the sum of all weight values ​​is 1.

[0044] Weighted fusion processing is performed according to the weights. In the specific implementation of this embodiment, the weighted fusion calculation formula is as follows: ,in The features are those after fusion, where n is the total number of modes. Let i be the fusion weights for the i-th mode. Let be the alignment feature of the i-th modality. After fusion, a feature filtering network consisting of a fully connected layer and a Sigmoid activation layer is constructed. The filtering threshold is set to 0.5, retaining high-contribution feature dimensions with activation values ​​greater than the threshold and removing redundant and invalid dimensions. Finally, a unified representation result of 1024-dimensional multimodal heterogeneous features is output.

[0045] The unified representation output is adapted to downstream tasks. This step applies the unified representation results to the cross-modal retrieval task and performs iterative optimization of the model based on task performance. First, the unified representation features of all samples are output and stored in the retrieval database to construct a multimodal feature index. When the cross-modal retrieval task is executed, the query sample is used to generate unified representation features using the method of this invention. The cosine similarity is calculated between the query sample and the unified representation features of all samples in the database, and the 10 results with the highest similarity are returned.

[0046] In the model iterative optimization phase, based on the cross-modal retrieval accuracy feedback, the parameters of the modality-specific adapting sub-network, the large model encoding parameters alignment mechanism, the loss weight, and the fusion weight are iteratively adjusted. In this embodiment, a total of 30 iterations of training are conducted, and the model loss value stabilizes below 0.05, achieving model convergence. The optimization process adopts a closed-loop mode of simulation and real-world evaluation to continuously improve the accuracy of the unified representation and its adaptability to downstream tasks.

[0047] In summary, this embodiment uses cross-modal retrieval as a practical application scenario, fully realizing the unified representation of four heterogeneous modalities: text, image, audio, and tactile sensor signals. Comparative experiments verify that the method of this invention comprehensively outperforms existing mainstream technical solutions in core performance. This method achieves deep semantic matching of features from different modalities through a dual semantic alignment mechanism, resulting in high accuracy in cross-modal retrieval results and stably meeting the practical needs of high-precision cross-modal retrieval. It exhibits excellent fine-grained feature preservation, completely retaining key details such as image texture, audio emotion, and sensor physical changes, effectively avoiding semantic loss caused by modality conversion. Inference efficiency is significantly improved, memory resource consumption is greatly reduced, and it can efficiently process complex data such as long texts and high-resolution images, adapting to the real-time operation requirements of practical scenarios. It possesses strong generalization and adaptation capabilities for niche modalities and scarce data scenarios, achieving stable feature representation without requiring a large amount of labeled data, significantly broadening the application scope of multimodal technology. By leveraging pre-trained large models, the data and time costs of model training are significantly reduced. The models are compatible with various hardware platforms such as servers and edge devices, enabling rapid integration with various downstream multimodal tasks. Engineering deployment is cost-effective and highly practical. Overall, the method of this invention has outstanding advantages in semantic alignment, fine-grained preservation, operational efficiency, generalization adaptation, and engineering deployment, and can stably support large-scale applications in complex multimodal scenarios such as smart healthcare, autonomous driving, and smart cities.

[0048] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A unified representation method for multimodal heterogeneous features based on large-model semantic encoding, characterized in that, The method includes the following steps: S1. Collect raw data from at least two different modalities, perform dedicated preprocessing for each modal data, extract the initial features of each modality and perform standardization processing to obtain a standardized heterogeneous initial feature set; S2. For the initial features of different modalities, construct mutually independent modality-specific adaptation subnetworks, map the initial features of each modality to the dimension that matches the semantic space of the large model, perform semantic enhancement processing on the fine-grained features of each modality, and obtain the semantically enhanced intermediate features of each modality. S3. Input the semantic enhancement intermediate features of each modality into the semantic coding layer of the pre-trained large model to perform deep semantic coding, introduce residual connections and normalization processing to optimize the coding results, and obtain the semantic coding features of each modality. S4. Based on the semantic encoding features of the text modality, a dual semantic alignment mechanism combining cross-modal attention alignment and contrastive learning alignment is constructed. Semantic alignment processing is performed on the semantic encoding features of each modality to obtain an aligned heterogeneous feature set. S5. Dynamically allocate fusion weights based on the semantic contribution of each modality alignment feature, perform weighted fusion, perform adaptive filtering on the fused features, and output a unified representation result of multimodal heterogeneous features with fixed dimensions. S6. Based on the performance feedback from downstream tasks, iteratively optimize the parameters of the modality-specific adaptor subnetwork, the large model encoding parameters, the alignment mechanism loss weights and fusion weights, and update the unified representation results.

2. The multimodal heterogeneous feature unified representation method based on large model semantic coding according to claim 1, characterized in that, Step S1, when performing dedicated preprocessing for each modality of data, includes the following steps: For the raw text modal data, text cleaning, word segmentation, stop word removal and entity resolution are performed, and the text data is converted into initial text features using a word embedding model; For the raw image modal data, size normalization, pixel value normalization and data augmentation processing are performed, and the initial visual features of the image are extracted using a visual encoder; For the original audio modal data, resampling, denoising and short-time Fourier transform processing are performed to extract the Mel frequency cepstral coefficients, time domain features and frequency domain features of the audio, and convert them into the initial audio features; For the raw sensor signal modal data, filtering, noise reduction, and normalization are performed to convert the physical signals into initial sensor features, and spatiotemporal alignment processing of multimodal data is performed.

3. The multimodal heterogeneous feature unified representation method based on large model semantic coding according to claim 2, characterized in that, The standardization process performed in step S1 uses the L2 normalization method to perform sample-by-sample norm normalization on the initial features extracted from each modality, unifying the norm of all initial feature vectors to a fixed value. The spatiotemporal alignment of multimodal data uses a combination of hardware timestamp synchronization and software time compensation to unify the time base of different modal data. The initial features of all modalities are stored in a two-dimensional tensor format. The first dimension of the tensor is the feature sequence length, and the second dimension of the tensor is the feature dimension of the corresponding modality. The high-resolution image uses sliding window segmentation and feature stitching to extract the initial visual features.

4. The multimodal heterogeneous feature unified representation method based on large model semantic coding according to claim 1, characterized in that, In step S2, each modality-specific adaptor subnetwork constructed adopts a combination structure of fully connected layers, normalization layers, and activation layers. The training parameters of each adaptor subnetwork are independent of each other, and the input of each adaptor subnetwork is the standardized initial features of the corresponding modality. The output dimension of each adaptor subnetwork is consistent with the input dimension of the semantic encoding layer of the pre-trained large model. The semantic enhancement intermediate features of the text modality are generated through a weighted fusion method, and the fusion formula is as follows: , in the formula To enhance the semantic meaning of the text, intermediate features are used, where α is the weight coefficient of the intermediate features. The value of α is determined based on the statistical results of the semantic contribution of the text modal features. To adapt intermediate features for text output, Semantic vectors are embedded into pre-trained words for the large model. The semantically enhanced intermediate features from non-textual modalities are generated through weighted fusion guided by cue words. The fusion formula is as follows: , in the formula For non-textual modal semantic enhancement intermediate features, β is the weight coefficient of the non-textual intermediate features, and the value of β is determined based on the requirements for preserving fine-grained features of the non-textual modality. For non-text adaptation, output intermediate features. The semantic vector generated to guide the prompt words.

5. The multimodal heterogeneous feature unified representation method based on large model semantic coding according to claim 4, characterized in that, The text modality weight coefficient α is determined through multiple sets of text feature semantic expression integrity tests. During the test, text data of different lengths and types are selected, and the semantic matching degree of the semantic enhancement intermediate features under different coefficients is statistically analyzed. The value with the highest semantic matching degree is selected as the final coefficient. The non-text modality weight coefficient β is determined through fine-grained feature retention rate tests. During the test, fine-grained features related to image texture, audio emotion, and sensor signal changes are collected, and the retention degree of fine-grained features under different coefficients is statistically analyzed. The value with the best retention degree is selected as the final coefficient.

6. The multimodal heterogeneous feature unified representation method based on large model semantic coding according to claim 1, characterized in that, Step S3 includes the following steps when performing deep semantic encoding: Configure the encoding parameters of the pre-trained large model, set the context window length, number of encoding layers, number of attention heads and dropout probability, and fix the parameters of the backbone network of the pre-trained large model; For semantic enhancement intermediate features of text modalities, they are directly input into the semantic encoding layer of the pre-trained large model to perform deep semantic encoding processing; For semantic enhancement intermediate features of non-text modalities, they are concatenated with the semantic prompt text of the corresponding modalities, and then input into the semantic encoding layer of the pre-trained large model to perform deep semantic encoding processing; LayerNorm normalization is used to process the encoded output results, and residual connections are introduced to fuse the semantically enhanced intermediate features with the encoded output features to obtain the semantic encoding features of each modality.

7. The multimodal heterogeneous feature unified representation method based on large model semantic coding according to claim 1, characterized in that, In step S4, the text modality semantic encoding features used are cross-modal alignment baseline features. Cross-modal attention alignment employs a multi-head attention mechanism, using non-text modality semantic encoding features as query terms and text modality semantic encoding features as key and value terms. The semantic correlation between each modality feature and the baseline feature is calculated. Contrastive learning alignment uses the InfoNCE loss function to optimize the feature space distribution. The loss function formula is as follows: , in the formula To compare the learning loss values, N is the number of positive sample pairs and M is the number of negative sample pairs. This is the function for calculating feature similarity. For the current modality encoding features, For features corresponding to the same semantic modality, Encoding features for negative samples The temperature coefficient is determined based on the requirement of smoothing the distribution in the characteristic space.

8. The multimodal heterogeneous feature unified representation method based on large model semantic coding according to claim 7, characterized in that, The positive sample pairs in the contrastive learning loss function are combinations of encoded features from different modalities within the same semantic scene, while the negative sample pairs are combinations of encoded features from arbitrary modalities within different semantic scenes. (Temperature coefficient) The temperature coefficient is used to control the distribution range of similarity values. The value is determined by the clustering effect test in the feature space. During the test, the temperature coefficient value is adjusted, the cluster compactness of the same semantic features and the degree of separation of different semantic features are observed, and the value with the best clustering effect is selected as the final temperature coefficient.

9. The multimodal heterogeneous feature unified representation method based on large model semantic coding according to claim 1, characterized in that, Step S5 includes the following steps when performing weighted fusion and adaptive filtering: Cosine similarity is used to calculate the semantic similarity between each modality alignment feature and the text modality baseline feature. The similarity calculation formula is as follows: , in the formula Let A be the feature cosine similarity, A be the modality alignment feature to be calculated, and B be the text modality baseline alignment feature. and These are the L2 norms of the corresponding features; Based on the cosine similarity results, the fusion weights for each modality are generated by normalization, and the sum of all weight values ​​is 1; Weighted fusion is performed according to the fusion weights, and the fusion formula is as follows: , in the formula The features are those after fusion, where n is the total number of modes. Let i be the fusion weights for the i-th mode. The alignment feature is for the i-th mode; A feature selection network consisting of a fully connected layer and a Sigmoid activation layer is constructed. Feature dimensions are selected according to a preset threshold, and a unified representation result of multimodal heterogeneous features with fixed dimensions is output.

10. The multimodal heterogeneous feature unified representation method based on large model semantic coding according to claim 1, characterized in that, In step S2, the image modality adapter subnetwork introduces an attention mechanism to strengthen the weights of fine-grained features, the audio modality adapter subnetwork performs feature mapping in conjunction with emotion features, and the sensor signal modality adapter subnetwork injects physical constraints to complete feature transformation. All non-text modality adapter subnetworks convert their output features into discrete feature units that match the semantic space of the large model. The dimension of the discrete feature units is consistent with the dimension of the output features of the text modality adapter subnetwork. During the semantic enhancement process, the fine-grained feature dimensions related to texture, emotion, and physical changes of non-text modalities are preserved, and the numerical distribution range of fine-grained features is not compressed.

Images