Cross-domain multi-modal data automatic association mapping and fusion identification method
By using encoder and decoder networks in multimodal data for comparison learning and reconstruction, combined with greedy matching algorithm, the data aligned caused by cross-domain perspective differences is solved, and the accuracy of multimodal fusion recognition is improved and Model learning effect.
Patent Information
- Application Number
- CN202411060669.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-05
- Publication Date
- 2024-11-26
AI Technical Summary
In the existing multimodal fusion recognition technology, the problem of modal error association caused by cross-domain perspective differences and data transmission delay, especially in cloud-edge collaborative cross-domain target perception tasks, resulting in data misalignment and affecting the model learning effect.
Two encoder networks and two decoder networks are initialized, and the representation is extracted in the latent space by comparing the learning loss function, and the original information is maintained by using the reconstruction loss function, and feature splicing and matching are combined with the greedy matching algorithm to improve the alignment representation. accuracy.
It improves the accuracy of model recognition, reduces the impact of wrongly aligned samples, maintains the original information of modal features, and enhances the ability to identify unknown alignment relationships.
Smart Images

Figure CN119026004A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of multimodal fusion, and in particular to a method for automatic association mapping and fusion recognition of cross-domain multimodal data. Background Art
[0002] "Modality" refers to the form of information presentation, such as vision, text, voice, etc. In the field of artificial intelligence, multimodal fusion recognition combines data from different modalities, such as text, images, sound, and video, and uses common information and complementary information in multiple modalities to learn the fusion representation between modalities. The fusion representation is used to extract the associated information in the modalities to identify the identity, type, and behavioral intention of the target. In the prior art, the recognition of targets often relies on information from a single modality, but a single modality is one-sided and has limitations. Data from different modalities have their own characteristics and advantages. By fusing them for recognition, the ability to recognize targets can be improved and more accurate results can be obtained. However, there are still challenges in effectively realizing multimodal fusion and improving recognition effects.
[0003] At present, multimodal fusion recognition relies on the correct association of modalities. However, there are problems such as large cross-domain perspective differences and transmission delays in data collection. For example, when a photoelectric sensor and sound waves detect a target at the same time, the photoelectric speed is inconsistent, which leads to misalignment. In addition, the time difference is affected by the distance, which will lead to the above-mentioned modal misassociation phenomenon. In the cloud-edge collaborative cross-domain target perception task, each edge device / multiple agents collect data separately and collect and aggregate them to the cloud. Due to the differences in the perspectives of data collection and the different locations of each device, the cloud data is not aligned. It should be emphasized that since the "clocks" of multiple agents / edge devices are different, they cannot be processed by traditional calibration methods. This data misassociation seriously affects the learning of the model.
[0004] The multimodal partial view alignment problem is the key to the multimodal mis-association problem. In the multimodal partial view alignment problem, some views are known to be aligned, while the other views have unknown alignment relationships. How to perform multimodal clustering in the case of misaligned views is a key issue that needs to be solved urgently. The core difficulty of this problem is to use a dataset with misaligned samples to avoid the impact of misaligned samples as much as possible, but the alignment information is usually unknown.
[0005] At present, for the problem of partial view alignment, the existing technology mainly uses partial data with known alignment to learn a neural network for alignment. After learning, the data is aligned through the neural network for alignment. After training, the non-aligned samples are input into the network to obtain aligned features. For example, the partial view alignment clustering PVC uses the clean sample data of the aligned part to train a differentiable Hungarian matching module to learn the alignment information between different modalities of the data. After learning the alignment information, all the data is learned through the network to learn the correspondence between samples of different modalities. By using the Hungarian matching module, samples with unknown alignment relationships can be aligned to obtain alignment representations, but this method has high sample-level alignment complexity. The partial view alignment representation learning method MvCLN with noise robust contrast loss function adopts contrastive learning and a loss function with slow optimization and reverse optimization. By constructing positive and negative sample pairs, the influence of false positive samples and false negative samples on training is weakened. However, the robust loss function is prone to underfitting, and the absence of reconstruction loss will cause the extracted features to have information loss compared to the original features. Summary of the invention
[0006] To solve the above problems, the present invention provides a method for automatic association mapping and fusion recognition of cross-domain multimodal data. The specific technical solution is as follows:
[0007] S1: Initialize two encoder networks and two decoder networks;
[0008] S2: Build a cross-domain multimodal dataset and randomly sample a batch of samples from the dataset;
[0009] S3: Extract the representation of all samples in the latent space through the encoder network;
[0010] S4: Compute the loss function of the representation in the latent space through contrastive learning loss function;
[0011] S5: concatenate the representations in the latent space and decode them using two decoder networks respectively;
[0012] S6: Calculate the reconstruction loss function through the decoded samples and the input samples;
[0013] S7: sum the reconstructed loss function and back propagate it to update the network parameters until the model converges;
[0014] S8: Input the cross-domain multimodal data into the trained model to obtain the output results.
[0015] Furthermore, the input dimensions of the two encoders correspond to the dimensions of the two modal data, and the output dimensions of the two decoders correspond to the dimensions of the two modal data, respectively.
[0016] Furthermore, the decoder input dimension is equal to twice the encoder output dimension.
[0017] Furthermore, each sample of the cross-domain multimodal dataset consists of features of two modalities.
[0018] Furthermore, the features of the two modalities are 20-dimensional GIST features and 59-dimensional PHOG features, respectively.
[0019] Furthermore, the loss function of the representation in the latent space is as follows:
[0020]
[0021] Among them, B represents the number of samples randomly sampled from the data set, N represents the number of samples in the data set, sim represents the similarity function, i represents the counting parameter of a batch of samples randomly sampled from the data set, and j represents the counting parameter of the samples in the data set. and It means that the data of the i-th sample of two modalities is extracted through the encoder network, It represents the representation of the data of the jth sample of the v2 modality extracted through the encoder network in the latent space.
[0022] Furthermore, the calculation and reconstruction loss function is specifically expressed as follows:
[0023]
[0024]
[0025] in, It means randomly sampling the data of the i-th sample of the v-th modality in a batch of samples from the data set. Represents the data decoded by the decoder corresponding to the modality v, and d represents the number of samples in this batch of samples randomly sampled from the dataset.
[0026] Furthermore, the network parameters are updated as follows:
[0027] The gradient is obtained at each network node, and each network node updates the network weight according to the learning rate of the current node gradient.
[0028] Furthermore, the learning rate is 0.001.
[0029] Furthermore, after step S7, the method further includes:
[0030] Based on the collected sample data, it is divided into training set and test set in a 1:1 ratio, and the trained model is used to extract the features of the training set and test set respectively;
[0031] Train the support vector machine model using the training set;
[0032] For the test set features, a greedy matching algorithm is used for matching, and the matched features are concatenated according to the matching relationship to obtain the concatenated representation;
[0033] Testing support vector machines using the concatenated representations.
[0034] The beneficial effects of the present invention are as follows:
[0035] The present invention uses an encoder to obtain the representation of features in the latent space, performs cross-view comparative learning in the latent space, and uses a decoder of each modality to reconstruct the fused representation to maintain the original information of the modality, thereby improving the accuracy of model recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 It is a schematic flow chart of the method of the present invention.
[0037] Figure 2 It is a schematic diagram of the training network structure of the present invention.
[0038] Figure 3 It is a schematic diagram of the inference network structure of the present invention. DETAILED DESCRIPTION
[0039] The following description clearly and completely describes the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0040] In the description of the embodiments of the present invention, it should be noted that the indicated orientation or positional relationship is based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship in which the invention product is usually placed when in use, or the orientation or positional relationship commonly understood by those skilled in the art, or the orientation or positional relationship in which the invention product is usually placed when in use, which is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first" and "second" are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.
[0041] In the description of the embodiments of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a direct connection, or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0042] Example 1
[0043] Embodiment 1 of the present invention discloses a method for automatic association mapping and fusion recognition of cross-domain multimodal data, such as Figure 1 As shown, the specific steps are as follows:
[0044] S1: Initialize two encoder networks and two decoder networks;
[0045] Among them, the input dimensions of the two encoders correspond to the dimensions of the two modal data respectively, and the output dimensions of the two decoders correspond to the dimensions of the two modal data respectively.
[0046] S2: Build a cross-domain multimodal dataset and randomly sample a batch of samples from the dataset;
[0047] Multimodal data refers to embeddings of images, text, etc.
[0048] In this embodiment, the cross-domain multimodal dataset has 15 categories and 4485 samples, each of which consists of features of two modalities. In order to simulate the misalignment phenomenon, 50% of the alignment relationships in the dataset are randomly disrupted.
[0049] The features of the two modalities are respectively 20-dimensional GIST features and 59-dimensional PHOG features.
[0050] Specifically, the cross-domain multimodal dataset is represented as Among them, N is the total number of samples, v1 and v2 represent two modes, Represents the data of the i-th sample of the v-th modality in a batch of samples randomly sampled from the data set.
[0051] As a preferred embodiment, 1024 samples (each sample contains features of two modes) are randomly sampled from the data set as a batch of samples, denoted as
[0052] In this embodiment, the input dimension of the decoder is equal to twice the output dimension of the encoder;
[0053] Specifically, the input dimensions of the two encoders are 20 and 59 respectively, the output dimensions of the two decoders are 20 and 59 respectively, and the input dimensions of the decoders are 40 and 118 respectively.
[0054] S3: For each modality, for each feature Through the encoder network f v Extract the representation of all samples in the latent space like Figure 3 shown.
[0055] S4: The loss function of the representation in the latent space is calculated by contrastive learning loss function, as follows:
[0056]
[0057] Among them, B represents the number of samples randomly sampled from the data set, N represents the number of samples in the data set, sim represents the similarity function, i represents the counting parameter of a batch of samples randomly sampled from the data set, and j represents the counting parameter of the samples in the data set. and It means that the data of the i-th sample of two modalities is extracted through the encoder network, It represents the representation of the data of the jth sample of the v2 modality extracted through the encoder network in the latent space;
[0058] sim represents the similarity function, t is a parameter.
[0059] S5: Concatenate the representations in the latent space, i.e. And through the decoder g of each modality v Decode and reconstruct to get
[0060] S6: Calculate the reconstruction loss function through the decoded samples and the input samples, as follows:
[0061]
[0062]
[0063] in, It means randomly sampling the data of the i-th sample of the v-th modality in a batch of samples from the data set. Represents the data decoded by the decoder corresponding to the modality v, and d represents the number of samples in this batch of samples randomly sampled from the dataset.
[0064] S7: Sum the reconstructed loss function and back propagate it to update the network parameters until the model converges, such as Figure 2 shown.
[0065] The updating of network parameters is as follows:
[0066] Obtain the gradient at each network node, and each network node updates the network weight according to the learning rate of the current node gradient;
[0067] Specifically, the learning rate is 0.001.
[0068] According to the collected samples, they are divided into training set and test set in a 1:1 ratio, and the trained model is used to extract the features of the training set and test set respectively;
[0069] Use the training set features to train the support vector machine model; the support vector machine is a supervised classification model that achieves classification by finding the optimal boundary between different categories in the data set;
[0070] For the test set features, a greedy matching algorithm is used for matching, that is, the representation of the latent space of modality v1 is re-paired with the representation of the latent space of the nearest modality v2, and the matched features are concatenated to obtain the concatenated representation;
[0071] Testing support vector machines using the concatenated representations.
[0072] S8: Input the cross-domain multimodal data into the trained model to obtain the output results.
[0073] The present invention is not limited to the above-mentioned specific embodiments, but extends to any new features or any new combination disclosed in this specification, as well as any new method or process steps or any new combination disclosed.
Claims
1. A method for automatic association mapping and fusion recognition of cross-domain multimodal data. It is characterized in that include: S1: Initialize two encoder networks and two decoder networks; S2: Build a cross-domain multimodal dataset and randomly sample a batch of samples from the dataset; S3: Extract the representation of all samples in the latent space through the encoder network; S4: loss function for computing representations in latent space via contrastive learning loss function; S5: concatenate the representations in the latent space and decode them using two decoder networks respectively; S6: Calculate the reconstruction loss function through the decoded samples and the input samples; S7: sum the reconstructed loss function and back propagate it to update the network parameters until the model converges; S8: Input the cross-domain multimodal data into the trained model to obtain the output results.
2. According to claim 1, the cross-domain multimodal data automatic association mapping and fusion recognition method, It is characterized in that The input dimensions of the two encoders correspond to the dimensions of the two modal data, and the output dimensions of the two decoders correspond to the dimensions of the two modal data.
3. The cross-domain multimodal data automatic association mapping and fusion recognition method according to claim 2, It is characterized in that The decoder input dimension is equal to twice the encoder output dimension.
4. The cross-domain multimodal data automatic association mapping and fusion recognition method according to claim 3, It is characterized in that Each sample of the cross-domain multimodal dataset consists of features of two modalities.
5. According to claim 4, the cross-domain multimodal data automatic association mapping and fusion recognition method, It is characterized in that The features of the two modalities are respectively 20-dimensional GIST features and 59-dimensional PHOG features.
6. The cross-domain multimodal data automatic association mapping and fusion recognition method according to claim 1, It is characterized in that The loss function of the representation in the latent space is as follows: Among them, B represents the number of samples randomly sampled from the data set, N represents the number of samples in the data set, sim represents the similarity function, i represents the counting parameter of a batch of samples randomly sampled from the data set, and j represents the counting parameter of the samples in the data set. and It means that the data of the i-th sample of two modalities is extracted through the encoder network, Represents v extracted through the encoder network 2 The representation of the data of the jth sample of the modality in the latent space.
7. The cross-domain multimodal data automatic association mapping and fusion recognition method according to claim 1, It is characterized in that The calculation and reconstruction loss function is specifically expressed as follows: in, It means randomly sampling the data of the i-th sample of the v-th modality in a batch of samples from the data set. Represents the data decoded by the decoder corresponding to the modality v, and d represents the number of samples in this batch of samples randomly sampled from the dataset.
8. The cross-domain multimodal data automatic association mapping and fusion recognition method according to claim 1, It is characterized in that The updating of network parameters is as follows: The gradient is obtained at each network node, and each network node updates the network weight according to the learning rate of the current node gradient.
9. The cross-domain multimodal data automatic association mapping and fusion recognition method according to claim 8, It is characterized in that The learning rate is 0.
001.
10. The cross-domain multimodal data automatic association mapping and fusion recognition method according to any one of claims 1 to 9, It is characterized in that After step S7, the method further includes: Based on the collected sample data, it is divided into training set and test set in a 1:1 ratio, and the trained model is used to extract the features of the training set and test set respectively; Train the support vector machine model using the training set; For the test set features, a greedy matching algorithm is used for matching, and the matched features are concatenated according to the matching relationship to obtain the concatenated representation; Testing support vector machines using the concatenated representations.
Citation Information
Cited By
Cross-domain multi-modal data automatic association mapping and fusion recognition method
WO2026032240A1
