A Multimodal Data Processing Method Based on Modal-Specific Feature Compensation
By using modality-specific feature compensation, the problem of high computational complexity and low accuracy caused by the modal differences between images and text is solved. By employing dual-stream feature extraction, modality compensation, and channel fusion modules, efficient multimodal data feature representations are generated, improving the accuracy of cross-modal data matching.
Patent Information
- Application Number
- CN202611592794.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-15
- Publication Date
- 2026-11-17
AI Technical Summary
Existing multimodal data processing methods, when mitigating the modal differences between images and text, generate networks with complex structures and unstable training. The generated fake content may introduce data noise, and the methods do not adequately handle modal differences in the channel or semantic dimensions, resulting in high computational complexity.
A modality-specific feature compensation method is adopted. Image and text features are extracted separately by a dual-stream feature extraction module. A modality compensation module is constructed to generate compensation features. Feature decomposition and recombination are performed using a multimodal channel fusion module. The network is optimized by combining multimodal triplet loss and feature consistency loss.
This approach achieves the generation of more closely aligned and more discriminative multimodal data feature representations while reducing computational complexity, thereby improving the accuracy of cross-modal data retrieval and matching.
Smart Images

Figure CN123116176A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing technology, and more specifically to a multimodal data processing method based on modality-specific feature compensation. Background Technology
[0002] Multimodal data matching and recognition is a key technology in information processing, aiming to accurately retrieve and match the identity of the same target object from input data of different modalities. Traditional methods are mainly geared towards single-modal (e.g., image-image) retrieval, relying on the appearance features of the target. However, in real-world data processing applications such as security monitoring and identity authentication, users often find it difficult to provide reference images of the target object and prefer to use natural language descriptions to query the target. This has given rise to the image-text cross-modal data matching task. This task constructs a modality-shared semantic representation by jointly learning image data and text descriptions, achieving target identity matching for inputs from different modalities. Its flexibility and interactivity are significantly better than traditional retrieval systems that rely solely on a single modality. However, the inherent modal differences between images and text become a core challenge: image data presents visual attributes such as color, texture, and shape in the form of pixel matrices, while text is a discrete sequence of symbols that encodes abstract semantic attributes (such as category, behavior, personal belongings, etc.). The fundamental differences in data representation and semantic granularity make the unified representation and accurate matching of multimodal data extremely difficult.
[0003] Existing data processing methods have explored various approaches to mitigate the disparity between text and image modalities. Some studies employ generative strategies, narrowing the heterogeneity gap by generating intermediate modal representations (such as converting text into attribute vectors or generating pseudo-image features). However, generative networks are complex and training is unstable, and generated pseudo-content may introduce data noise. Furthermore, eigenvalue decomposition methods have been used to separate modality-shared information from modality-specific information to improve feature discriminative power. Despite significant progress, existing research largely focuses on aligning global features, and its fine-grained handling of modality differences along channel or semantic dimensions remains insufficient. Additionally, some methods rely on complex generative networks, leading to a heavy training burden on the data processing system. Summary of the Invention
[0004] The purpose of this invention is to overcome the shortcomings of existing technologies and propose a multimodal data processing method based on modality-specific feature compensation. Taking semantic channels as the starting point, this method achieves effective compensation and unified data representation of heterogeneous image and text information by collaboratively learning modality-shared features, image-specific features, text-specific features, and modality compensation features, thereby reducing computational complexity while maintaining high performance.
[0005] The technical solution adopted by this invention to solve its technical problem is: A multimodal data processing method based on modality-specific feature compensation includes: Image data and text description data are used together as multimodal data input; A dual-stream feature extraction module is constructed, and an asymmetric coding architecture is used to extract features from the image data and the text description data. The image stream uses a ResNet-50 convolutional neural network to extract visual features, while the text stream uses a Transformer encoder to extract semantic features. The text semantic features are expanded into spatial feature maps through broadcast reshaping to adapt to subsequent multimodal data interaction processing, thus obtaining shared features of the image modality and shared features of the text modality. A modal compensation module is constructed, taking the shared features of the image modality and the shared features of the text modality as input, and using two independent modal compensation branches with non-shared parameters, inferring the missing semantic information of the text modality from the image modal features, and inferring the missing visual information of the image modality from the text modal features, respectively generating image compensation features and text compensation features to compensate for the information asymmetry between image and text data; A multimodal channel fusion module is constructed. A channel mask is generated through a shared channel attention network. The channel mask is used to decompose and reorganize the common features of the image modality and the common features of the text modality in terms of channel dimension. Modality-shared features and modality-specific features are extracted. The inverse code of the channel mask is used to extract the discriminative information unique to each modality, thereby realizing the separation of redundant data information and the enhancement of discriminative features. The modality-shared features, modality-specific features, image compensation features, and text compensation features are fused to generate a unified multimodal data feature representation. The similarity between the query sample and the database sample is calculated based on the unified feature representation, and the cross-modal data matching result is output.
[0006] Furthermore, the processing procedure of the dual-stream feature extraction module includes: ResNet-50 was used as the backbone network, and CONV layers with non-shared parameters were used in the shallow layers of the network to extract image modal features and text modal features respectively. The extracted image modal features and text modal features are input into the STAGE1 to STAGE4 layers of ResNet-50, and the common feature representations of the two modalities are learned.
[0007] Furthermore, the processing procedure of the modal compensation module includes: The image compensation branch takes the common features of the image modalities as input to generate image compensation features carrying textual semantic information in order to make up for the lack of abstract semantic discrimination information in the image data; The text compensation branch takes the common features of the text modalities as input to generate text compensation features carrying visual information in order to make up for the missing visual discrimination information in the text data; The image compensation branch has the same structure as the text compensation branch, but their parameters are not shared.
[0008] Furthermore, the process of generating the image compensation features includes: The common features of the image modalities F v ∈R C×H×W By inputting the global max pooling layer and the global average pooling layer respectively, global contextual information at different levels in the feature map is captured, resulting in global max pooling features and global average pooling features: A max = Conv 1×1 (ReLU(Conv 1×1 GMP F v )))); A avg = Conv 1×1 (ReLU(Conv 1×1 (GAP) F v )))); in, H and W The spatial dimensions of the feature map. C For the number of channels, Conv 1×1 This indicates a convolutional layer with a kernel size of 1×1, where ReLU is the activation function, introducing non-linear transformation capability; The outputs of the two paths are summed element-wise and then activated by the Sigmoid function. s Generate channel attention weights M ∈R C : M = s ( A max + A avg ); The channel attention weights M Features shared with the image modality F v Element-wise multiplication is performed to recalibrate the original input features, resulting in an attention-weighted feature representation. F ' v ∈R C×H×W : F' v = M ☉ F v + F v ; Where ☉ represents element-wise multiplication; Attention-weighted feature representation F ' v The image is then fed into a feature projection network, which consists of global average pooling, fully connected layers, batch normalization, and ReLU activation function, to generate image compensation features. F c,v ∈R C' : F c,v = ReLU(BN(Linear(GAP( F ' v )))); Where GAP represents global average pooling, Linear represents a fully connected layer, and BN represents batch normalization.
[0009] Furthermore, the processing procedure of the multimodal channel fusion module includes: The common features of the image modalities F v ∈R C×H×W Shared features with the text modality F i ∈R C×H×W A common input is a channel attention network (CA) with shared parameters, which generates a channel mask M. * : M * = CA( F v , F i ); in, H and W The spatial dimensions of the feature map. C The channel attention network CA is a parameter-shared attention network, and M is the generated channel mask, where the value of each element is between 0 and 1, used to measure the contribution of the corresponding channel to the modality sharing information. Based on the generated channel mask M * Decompose the input features; Using the channel mask M * Extract modality-shared features between the two modalities: Fsh ,v = M* ☉ F v ; F sh ,i = M* ☉ F i ; in, F sh ,v Indicates image modality sharing features, F sh ,i ☉ represents text modality-shared features, and ☉ represents element-wise multiplication in the channel dimension; Using the channel mask M * one's complement 1-M * Extract mode-specific features from two modes: F sp ,v = (1- M* )☉ F v ; F sp ,i = (1- M* )☉ F i ; in, F sp ,v It represents specific features of an image modality, used to preserve unique discriminative information such as color and texture in image data; F sp ,i It represents specific features of text modality and is used to retain unique discriminative information such as combinations of abstract attributes in text data.
[0010] Furthermore, the method employs a multimodal triplet loss L c_tri and feature consistency loss L fc Jointly optimize the network: Total loss function L total Represented as: L total = L c_tri + l L fc ; in, l This is a hyperparameter used to adjust the weight of the feature consistency loss in the total loss.
[0011] Furthermore, the multimodal triplet loss L c_tri The specific calculation process includes: Sharing features across image modalities F sh ,v Sharing features with text modalitiesF sh ,i As a constraint, it brings the distance between multimodal positive sample pairs of the same target object closer, while pushing the distance between multimodal negative sample pairs of different target objects further apart; The multimodal triplet loss L c_tri It consists of two symmetrical sub-losses. The first term uses image features as the anchor, text features of the same identity as positive samples, and text features of different identities as negative samples. The second term uses text features as the anchor, image features of the same identity as positive samples, and image features of different identities as negative samples. ; in, P This indicates the number of target object identities selected in each training batch. Kv and Ki These represent the number of images and the number of text samples for each identity, respectively. α Indicates boundary parameters; The first item is the first r The first identity a Zhang Image Features F v r, a As anchors, textual features of the same identity F i r, p Positive samples, image features of different identities F v j, n Negative samples; The second item is the first r The first a Zhang's text features F i r, a Image features of the same identity, serving as anchor points. F v r, p For positive samples, text features of different identities F i j, n This is a negative sample.
[0012] Furthermore, the feature consistency loss L fc The specific calculation process includes: By constraining the distance between image-compensated features and text-specific feature centers, as well as the distance between text-compensated features and image-specific feature centers, the network is guided to learn specific information missing from each modality: ; in, P and KThese represent the number of target object identities selected in each training batch and the number of images corresponding to each identity, respectively. F c,v p, k , F c,i p, k The first p The first goal k Compensation features corresponding to the image and text C sp,v p , C sp,i p They are the first p The specific center of a target in both image and text modalities. C sp,v q , C sp,i q They are the first q The specific center of a target in both image and text modalities. D () represents the Euclidean distance, used to measure the difference between the compensated feature and a specific feature center. It can effectively constrain the compensated feature to move closer to the specific feature center of the corresponding mode. Its calculation formula is: ; The specific feature center C sp,v p and C sp,i p The mean value of modality-specific features for the same identity in the current training batch is obtained by calculating: ; ; in, F c,v p,k Indicates the first p The first identity k Modal-specific features of an image, F c,i p,k Indicates the first p The first identity k Modality-specific features of a text.
[0013] Technical effects of the present invention: Compared with existing technologies, the present invention provides a multimodal data processing method based on modality-specific feature compensation. This method proposes a modality-specific feature compensation network comprising three core modules: First, a dual-stream feature extraction module employs an asymmetric coding architecture. Visual features are extracted from the image stream using ResNet-50, and semantic features are extracted from the text stream using a Transformer encoder. The text features are then expanded into a spatial feature map through broadcast reshaping to adapt to subsequent multimodal data interaction processing. Second, a multimodal channel fusion module introduces a channel attention mechanism to generate a dynamic mask, decomposing features into modality-shared features and modality-specific features, achieving the separation of redundant data information and the enhancement of discriminative features. Finally, a modality compensation module, through channel attention and feature projection networks, infers the missing semantic information of the target modality from the source modality features, effectively compensating for information asymmetry between image and text data. The three modules work synergistically, enabling the network to focus on more discriminative semantic channel patterns, generating more closely aligned and more discriminative multimodal data feature representations, thereby significantly improving the accuracy of cross-modal data retrieval and matching. Attached Figure Description
[0014] Figure 1 This is a diagram of the overall architecture of the modality-specific feature compensation network of the present invention; Figure 2 This is a structural diagram of the modal compensation module of the present invention. Detailed Implementation
[0015] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.
[0016] Example 1: like Figure 1 As shown in the figure, this embodiment relates to a multimodal data processing method based on modality-specific feature compensation, which is implemented based on a modality-specific feature compensation network. The modality-specific feature compensation network achieves effective compensation and unified feature representation of multimodal heterogeneous information by collaboratively learning modality-shared features, modality-specific features and modality-compensated features. It includes a dual-stream feature extraction module, a modality compensation module and a multimodal channel fusion module.
[0017] The multimodal data processing method based on modality-specific feature compensation includes the following steps: Step S1: Input both image data and text description data as multimodal data.
[0018] Step S2: Construct a dual-stream feature extraction module, such as... Figure 1As shown, an asymmetric coding architecture is used to extract features from the image data and the text description data. The image stream uses a ResNet-50 convolutional neural network to extract visual features, and the text stream uses a Transformer encoder to extract semantic features. The text semantic features are then expanded into a spatial feature map through broadcast reshaping to adapt to subsequent multimodal data interaction processing, thereby obtaining shared features of the image modality and shared features of the text modality.
[0019] Specifically, by employing a shallow-layer parameter-non-sharing and deep-layer parameter-sharing approach, common semantic features of the two modalities are extracted while preserving modal information. First, the CONV layer of a parameter-non-sharing ResNet-50 is used to extract modal features for both modalities. Then, the extracted modal features are input into STAGE1 to STAGE4 layers of a parameter-sharing ResNet-50 to learn the common features between the two modalities. F v , F i The specific process can be represented as follows: F v = i sh ( i sp v ( X v )); F i = i sh ( i sp i ( X i )); in, X v and X i These represent image and text inputs, respectively. i sp v and i sp i For the CONV layer of ResNet-50 that does not share parameters, i sh STAGE1 to STAGE4 of ResNet-50 with shared parameters.
[0020] Step S3: Construct a modal compensation module. Using the shared image modal features and the shared text modal features as input, employ two independent modal compensation branches with non-shared parameters. This involves inferring the missing semantic information of the text modality from the image modal features and inferring the missing visual information of the image modality from the text modal features, generating image compensation features and text compensation features respectively to compensate for the information asymmetry between image and text data. Specifically, the shared image modal features... F v Generate image compensation features carrying textual semantic information F c,v To compensate for the lack of abstract semantic discriminative information in images (such as attribute combinations in text descriptions); using shared features of text modalities. F i Generate text compensation features carrying visual information F c,i This is to compensate for the lack of visual discriminative information in the text (such as color details and texture distribution). Its specific design is as follows: Figure 2 As shown. The two modal compensation branches have the same structure; therefore, this embodiment uses the generation of image compensation features from shared image modal features as an example to illustrate its structure and calculation process in detail. This branch uses shared image modal features... F v ∈R C×H×W As input, the discriminative channels of image modality interest are first extracted through channel attention. Global max pooling and global average pooling are then performed to capture global contextual information at different levels in the feature map, resulting in global max pooling features and global average pooling features: A max = Conv 1×1 (ReLU(Conv 1×1 GMP F v )))); A avg = Conv 1×1 (ReLU(Conv 1×1 (GAP) F v )))); in, H and W The spatial dimensions of the feature map. C For the number of channels, Conv 1×1 This indicates a convolutional layer with a kernel size of 1×1, using ReLU as the activation function to introduce non-linear transformation capability; the outputs of the two paths are element-wise summed and then activated by the Sigmoid function. s Generate channel attention weights M ∈RC : M = s ( A max + A avg ); This weighting is essentially a channel selection mechanism that adaptively emphasizes feature channels relevant to the target modality while suppressing irrelevant or interfering channels. Subsequently, the channel attention weights are... M Features shared with the image modality F v Element-wise multiplication is performed to recalibrate the original input features, resulting in an attention-weighted feature representation. F ' v ∈R C×H×W : F ' v = M ☉ F v + F v ; Where ☉ represents element-wise multiplication; Attention-weighted feature representation F ' v The image is then fed into a feature projection network, which consists of global average pooling, fully connected layers, batch normalization, and ReLU activation function, to generate image compensation features. F c,v ∈R C' : F c,v = ReLU(BN(Linear(GAP( F ' v )))); Here, GAP represents global average pooling, Linear represents a fully connected layer, and BN represents batch normalization. Through these transformations, the network can generate textual modality semantic information missing from image modalities, thus achieving multimodal information compensation.
[0021] Step S4: Construct a multimodal channel fusion module. A channel mask is generated using a shared channel attention network. This channel mask is then used to decompose and reorganize the common features of the image modalities and the common features of the text modalities along the channel dimension. Modal shared features and modal-specific features are extracted. Furthermore, the inverse code of the channel mask is used to extract the discriminative information specific to each modality, thereby achieving the separation of redundant data information and the enhancement of discriminative features. Specifically, the common features of the image modalities...F v ∈R C×H×W Shared features with the text modality F i ∈R C×H×W A common input is a channel attention network (CA) with shared parameters, which generates a channel mask M. * : M * = CA( F v , F i ); in, H and W The spatial dimensions of the feature map. C For the number of channels, v and i The image modality and text modality are represented respectively. The channel attention network CA is a parameter-shared attention network, and M is the generated channel mask. The value of each element is between 0 and 1, which is used to measure the contribution of the corresponding channel to the modality-shared information. Based on the generated channel mask M * The input features are decomposed. First, the channel mask M is used. * Extract modality-shared features between the two modalities: F sh ,v = M* ☉ F v ; F sh ,i = M* ☉ F i ; in, F sh ,v Indicates image modality sharing features, F sh ,i ☉ represents text modality-shared features, and ☉ represents element-wise multiplication in the channel dimension; Then use the channel mask M * one's complement 1-M * Extract mode-specific features from two modes. Inverse mask 1−M * By focusing on the modality-specific channels and capturing modality-specific discriminative information, such as color and texture features in images and abstract attribute combinations in text, we can preserve the unique discriminative cues of each modality. This process can be represented as: F sp ,v = (1- M* )☉ F v ; F sp ,i = (1- M* )☉ F i ; in, F sp ,v It represents specific features of an image modality, used to preserve unique discriminative information such as color and texture in image data; F sp ,i It represents specific features of text modality and is used to retain unique discriminative information such as combinations of abstract attributes in text data.
[0022] Step S5: Fuse the modality-shared features, modality-specific features, image compensation features, and text compensation features to generate a unified multimodal data feature representation. Calculate the similarity between the query sample and the database sample based on the unified feature representation and output the cross-modal data matching result.
[0023] To effectively mitigate modal differences in cross-modal data matching and achieve more accurate multimodal data retrieval and matching, this chapter proposes multimodal triplet loss and feature consistency loss. The two loss methods work together to optimize the network, reducing modal differences while improving inter-class discriminability.
[0024] (1) Multimodal triplet loss In cross-modal data matching tasks, due to the significant differences in information representation between image and text modalities, traditional triplet loss is insufficient to effectively address the inconsistency in feature distribution between modalities. To address this challenge, this embodiment proposes a multimodal triplet loss based on the triplet loss, which addresses the issue of shared features between image modalities. F sh ,v Sharing features with text modalities F sh ,i Apply constraints. For simplicity, subscripts are omitted from the formula. sh ,by F v and F i These two features are represented respectively. The core function of this loss is to shorten the distance between multimodal positive sample pairs (matched image and text data pairs) of the same target object, while widening the distance between multimodal negative sample pairs of different target objects, thereby effectively promoting the alignment and discriminative learning of intermodal features.
[0025] The multimodal triplet loss L c_tri It consists of two symmetrical sub-losses. The first term uses image features as the anchor, text features of the same identity as positive samples, and text features of different identities as negative samples. The second term uses text features as the anchor, image features of the same identity as positive samples, and image features of different identities as negative samples. ; in, P This indicates the number of target object identities selected in each training batch. Kv and Ki These represent the number of images and the number of text samples for each identity, respectively. α Indicates boundary parameters; The first term of the formula begins with the first term. r The first identity a Zhang Image Features F v r, a As anchors, textual features of the same identity F i r, p Positive samples, image features of different identities F v j, n Negative samples; The second term of the formula begins with the first term. r The first a Zhang's text features F i r, a Image features of the same identity, serving as anchor points. F v r, p For positive samples, text features of different identities F i j, n This is a negative sample.
[0026] (2) Feature consistency loss To ensure that the features generated by the modality compensation module in this embodiment are semantically consistent with the actual modality-specific features, a feature consistency loss is proposed. This loss guides the network to learn the specific information missing in each modality by constraining the distances between image-compensated features and text-specific feature centers, and between text-compensated features and image-specific feature centers, thereby effectively mitigating modality discrepancies. The feature consistency loss is defined as follows: ; in, P and K These represent the number of target object identities selected in each training batch and the number of images corresponding to each identity, respectively. F c,v p, k , F c,i p, k The first p The first goal k Compensation features corresponding to the image and text C sp,vp , C sp,i p They are the first p The specific center of a target in both image and text modalities. C sp,v q , C sp,i q They are the first q The specific center of a target in both image and text modalities. D () represents the Euclidean distance, used to measure the difference between the compensated feature and a specific feature center. It can effectively constrain the compensated feature to move closer to the specific feature center of the corresponding mode. Its calculation formula is: ; The specific feature center C sp,v p and C sp,i p The mean value of modality-specific features for the same identity in the current training batch is obtained by calculating: ; ; in, F c,v p,k Indicates the first p The first identity k Modal-specific features of an image, F c,i p,k Indicates the first p The first identity k Modality-specific features of a text.
[0027] The total loss function is composed of the multimodal triplet loss Lc_tri and the feature consistency loss Lfc, and a hyperparameter is introduced for the feature consistency loss. l To adjust its weight in the total loss, the total loss function L total The mathematical expression for is defined as follows: L total = L c_tri + l L fc ; in, l This is a hyperparameter used to adjust the weight of the feature consistency loss in the total loss.
[0028] This invention employs a dual-stream ResNet-50 network as its backbone to better mitigate multimodal feature differences. By designing a dual-stream feature extraction module, this invention uses a non-shared parameter network to extract modal features from images and text separately in the shallow layers of the network, while sharing parameters are used in subsequent layers to obtain features common to both modalities. A modality compensation module is designed to generate semantic information of another modality missing from each modality at the feature level, thus compensating for information asymmetry between images and text. Two modality compensation modules with non-shared parameters are used for image and text modalities respectively, generating image compensation features from image features and text compensation features from text features to supplement the missing modality discrimination information in the features of this modality. In this way, the differences between modalities are effectively bridged, improving the consistency and completeness of feature representation. This invention designs a multimodal channel fusion module to decompose and reorganize the extracted features along the channel dimension. A channel mask is generated through a shared channel attention, and the mask is used to extract modality-shared features. Attention inverse codes are used to extract modality-specific features, thereby separating the modality-shared information of the two modalities from their respective unique discriminative information. This achieves effective separation of redundant data information and targeted enhancement of discriminative features.
[0029] The above-described specific embodiments are merely specific examples of the present invention. The patent protection scope of the present invention includes, but is not limited to, the above-described specific embodiments. Any appropriate changes or modifications made by a person skilled in the art that conform to the claims of the present invention should fall within the patent protection scope of the present invention.
Claims
1. A multimodal data processing method based on modality-specific feature compensation, characterized in that, include: Image data and text description data are used together as multimodal data input; A dual-stream feature extraction module is constructed, and an asymmetric coding architecture is used to extract features from the image data and the text description data. The image stream uses a ResNet-50 convolutional neural network to extract visual features, while the text stream uses a Transformer encoder to extract semantic features. The text semantic features are expanded into a spatial feature map through broadcast reshaping to obtain shared features of the image modality and shared features of the text modality. A modal compensation module is constructed, taking the shared features of the image modality and the shared features of the text modality as input, and using two independent modal compensation branches with non-shared parameters, inferring the missing semantic information of the text modality from the image modal features, and inferring the missing visual information of the image modality from the text modal features, respectively generating image compensation features and text compensation features to compensate for the information asymmetry between image and text data; A multimodal channel fusion module is constructed. A channel mask is generated through a shared channel attention network. The channel mask is used to decompose and reorganize the common features of the image modality and the common features of the text modality in terms of channel dimension. Modality-shared features and modality-specific features are extracted. The inverse code of the channel mask is used to extract the discriminative information unique to each modality, thereby realizing the separation of redundant data information and the enhancement of discriminative features. The modality-shared features, modality-specific features, image compensation features, and text compensation features are fused to generate a unified multimodal data feature representation. The similarity between the query sample and the database sample is calculated based on the unified feature representation, and the cross-modal data matching result is output.
2. The multimodal data processing method based on modality-specific feature compensation according to claim 1, characterized in that, The processing procedure of the dual-stream feature extraction module includes: ResNet-50 was used as the backbone network, and CONV layers with non-shared parameters were used in the shallow layers of the network to extract image modal features and text modal features respectively. The extracted image modal features and text modal features are input into the STAGE1 to STAGE4 layers of ResNet-50, and the common feature representations of the two modalities are learned.
3. The multimodal data processing method based on modality-specific feature compensation according to claim 1, characterized in that, The processing procedure of the modal compensation module includes: The image compensation branch takes the common features of the image modalities as input to generate image compensation features carrying textual semantic information in order to make up for the lack of abstract semantic discrimination information in the image data; The text compensation branch takes the common features of the text modalities as input to generate text compensation features carrying visual information in order to make up for the missing visual discrimination information in the text data; The image compensation branch has the same structure as the text compensation branch, but their parameters are not shared.
4. The multimodal data processing method based on modality-specific feature compensation according to claim 1, characterized in that, The process of generating the image compensation features includes: The common features of the image modalities F v ∈R C×H×W By inputting the global max pooling layer and the global average pooling layer respectively, global contextual information at different levels in the feature map is captured, resulting in global max pooling features and global average pooling features: A max = Conv. 1×1 (ReLU(Conv 1×1 (GMP( F v )))); A avg = Conv. 1×1 (ReLU(Conv 1×1 (GAP( F v )))); in, H and W The spatial dimensions of the feature map. C For the number of channels, Conv 1×1 This indicates a convolutional layer with a kernel size of 1×1, where ReLU is the activation function; The outputs of the two paths are summed element-wise and then activated by the Sigmoid function. σ Generate channel attention weights M ∈R C : M = σ ( A max + A avg ); The channel attention weights M Features shared with the image modality F v Element-wise multiplication is performed to recalibrate the original input features, resulting in an attention-weighted feature representation. F ' v ∈R C×H×W : F ' v = M ☉ F v + F v ; Where ☉ represents element-wise multiplication; Attention-weighted feature representation F ' v The image is then fed into a feature projection network, which consists of global average pooling, fully connected layers, batch normalization, and ReLU activation function, to generate image compensation features. F c,v ∈R C' : F c,v = ReLU(BN(Linear(GAP( F ' v )))); Where GAP represents global average pooling, Linear represents a fully connected layer, and BN represents batch normalization.
5. The multimodal data processing method based on modality-specific feature compensation according to claim 1, characterized in that, The processing procedure of the multimodal channel fusion module includes: The common features of the image modalities F v ∈R C×H×W Shared features with the text modality F i ∈R C×H×W A common input is a channel attention network (CA) with shared parameters, which generates a channel mask M. * : M * = CA( F v , F i ); in, H and W The spatial dimensions of the feature map. C The channel attention network CA is a parameter-shared attention network, and M is the generated channel mask, where the value of each element is between 0 and 1, used to measure the contribution of the corresponding channel to the modality sharing information. Based on the generated channel mask M * The input features are decomposed. Using the channel mask M * Extract modality-shared features between the two modalities: F sh ,v = M* ☉ F v ; F sh ,i = M* ☉ F i ; in, F sh ,v Indicates image modality sharing features, F sh ,i ☉ represents text modality-shared features, and ☉ represents element-wise multiplication in the channel dimension; Using the channel mask M * one's complement 1-M * Extract mode-specific features from two modes: F sp ,v = (1- M* )☉ F v ; F sp ,i = (1- M* )☉ F i ; in, F sp ,v It represents specific features of an image modality, used to preserve unique discriminative information such as color and texture in image data; F sp ,i It represents specific features of the text modality.
6. The multimodal data processing method based on modal-specific feature compensation according to claim 1, characterized in that, The method employs a multimodal triplet loss L c_tri and feature consistency loss L fc Jointly optimize the network: Total loss function L total Represented as: L total = L c_tri + λ L fc ; in, λ This is a hyperparameter used to adjust the weight of the feature consistency loss in the total loss.
7. The multimodal data processing method based on modality-specific feature compensation according to claim 6, characterized in that, The multimodal triplet loss L c_tri The specific calculation process includes: Sharing features across image modalities F sh ,v Sharing features with text modalities F sh ,i As a constraint, it brings the distance between multimodal positive sample pairs of the same target object closer, while pushing the distance between multimodal negative sample pairs of different target objects further apart; The multimodal triplet loss L c_tri It consists of two symmetrical sub-losses. The first term uses image features as the anchor, text features of the same identity as positive samples, and text features of different identities as negative samples. The second term uses text features as the anchor, image features of the same identity as positive samples, and image features of different identities as negative samples. ; in, P This indicates the number of target object identities selected in each training batch. Kv and Ki These represent the number of images and the number of text samples for each identity, respectively. α Indicates boundary parameters; The first item is the first r The first identity a Zhang Image Features F v r, a As anchors, textual features of the same identity F i r, p Positive samples, image features of different identities F v j, n Negative samples; The second item is the first r The first a Zhang's text features F i r, a Image features of the same identity, serving as anchor points. F v r, p For positive samples, text features of different identities F i j, n This is a negative sample.
8. The multimodal data processing method based on modality-specific feature compensation according to claim 6, characterized in that, The feature consistency loss L fc The specific calculation process includes: By constraining the distance between image-compensated features and text-specific feature centers, as well as the distance between text-compensated features and image-specific feature centers, the network is guided to learn specific information missing from each modality: ; in, P and K These represent the number of target object identities selected in each training batch and the number of images corresponding to each identity, respectively. F c,v p, k , F c,i p, k The first p The first goal k Compensation features corresponding to the image and text C sp,v p , C sp,i p They are the first p The specific center of a target in both image and text modalities. C sp,v q , C sp,i q They are the first q The specific center of a target in both image and text modalities. D () represents the Euclidean distance, used to measure the difference between the compensated feature and a specific feature center. It can effectively constrain the compensated feature to move closer to the specific feature center of the corresponding mode. Its calculation formula is: ; The specific feature center C sp,v p and C sp,i p The mean value of modality-specific features for the same identity in the current training batch is obtained by calculating: ; ; in, F c,v p,k Indicates the first p The first identity k Modal-specific features of an image, F c,i p,k Indicates the first p The first identity k Modality-specific features of a text.
