Cross-domain multi-modal data automatic association mapping and fusion recognition method

Abstract

Disclosed in the present invention is a cross-domain multi-modal data automatic association mapping and fusion recognition method, comprising: initializing two encoder networks and two decoder networks; constructing a cross-domain multi-modal data set, and randomly sampling a batch of samples from the data set; extracting representations of all the samples in a latent space by means of the encoder networks; calculating a loss function of the representations in the latent space by means of a contrastive learning loss function; concatenating the representations in the latent space, and respectively using the two decoder networks for decoding; calculating a reconstructed loss function by means of the decoded samples and input samples; performing summation on the reconstructed loss function and then performing back propagation to update network parameters until a model converges; and inputting cross-domain multi-modal data into a trained model. The present invention retains original information of modalities and improves the accuracy of model recognition.

Description

Automatic correlation mapping and fusion recognition method for cross-domain multi-modal data TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal fusion, and particularly relates to an automatic correlation mapping and fusion recognition method for cross-domain multi-modal data. BACKGROUND

[0002] "Modality" refers to the form of information, such as vision, text, speech, etc. In the field of artificial intelligence, multi-modal fusion recognition combines data from different modalities, such as text, images, sound, and video, etc., uses common information and complementary information in multiple modalities, learns fusion representations between modalities, and uses the fusion representations to extract related information in modalities to identify the identity, type, and behavior intention of a target. In the prior art, the identification of a target often relies on information from a single modality, but a single modality has one-sidedness and limitations. Different modalities of data have their own characteristics and advantages, and by fusing them for identification, the identification ability for a target can be improved to obtain more accurate results. However, there are still challenges in effectively implementing multi-modal fusion and improving the recognition effect.

[0003] Currently, multi-modal fusion recognition relies on correct correlation of modalities. However, there are problems such as large cross-domain perspective differences and transmission delays during data acquisition. For example, when an optical-electric sensor and a sound wave detect a target at the same time, the non-alignment will be caused due to the inconsistent speeds of the optical-electric sensor and the sound wave, and the time difference will be affected by the distance, which will lead to the above-mentioned modality mis-correlation phenomenon. In a cloud-edge collaborative cross-domain target perception task, each edge device / multi-agent collects data, and the data is collected and summarized to the cloud. Due to the perspective differences of the data collected by each device and the different locations, the data at the cloud is not aligned. It needs to be emphasized that since the "clocks" of the multi-agents / edge devices are different, the traditional calibration method cannot be used to handle it. This data mis-correlation seriously affects the learning of the model.

[0004] Multi-modal partial view alignment is a key problem of multi-modal mis-correlation. In the problem of multi-modal partial view alignment, a part of the views is known to be aligned, and another part of the views is unknown to be aligned. How to perform multi-modal clustering under the condition of view mis-alignment is a key problem that needs to be solved at present. The core difficulty of this problem is to use a data set with mis-aligned samples to avoid the influence of mis-aligned samples as much as possible, but the alignment information is usually unknown.

[0005] At present, for the partial view alignment problem, the prior art mainly uses known aligned partial data to learn a neural network for alignment. After learning is completed, the data is aligned through the neural network for alignment, and after training is completed, the unaligned samples are input into the network to obtain aligned features. For example, partial view alignment clustering PVC uses clean sample data of aligned parts to train a differentiable Hungarian matching module, learns alignment information between different modalities of data, and after learning the alignment information, learns the corresponding relationship between different modalities of data through the network, and through the Hungarian matching module, the samples with unknown alignment relationship can be aligned to obtain aligned representations, but this method has high sample-level alignment complexity. The partial view alignment representation learning method MvCLN with noise-robust contrastive loss function adopts contrastive learning and a loss function with slow optimization and reverse optimization, and by constructing positive and negative sample pairs, the influence of false positive and false negative samples on training is weakened, but the robust loss function is prone to underfitting, and the reconstruction loss will cause information loss of the extracted features compared with the original features. SUMMARY

[0006] To solve the above problems, the present application provides a cross-domain multi-modal data automatic association mapping and fusion recognition method, and the specific technical solutions are as follows:

[0007] S1: initialize two encoder networks and two decoder networks;

[0008] S2: build a cross-domain multi-modal dataset, and randomly sample a batch of samples from the dataset;

[0009] S3: extract the representation of all samples in the latent space through the encoder network;

[0010] S4: calculate the loss function of the representation in the latent space through the contrastive learning loss function;

[0011] S5: concatenate the representation in the latent space, and decode it using two decoder networks respectively;

[0012] S6: calculate the reconstruction loss function through the decoded samples and the input samples;

[0013] S7: sum the reconstruction loss function and back-propagate to update the network parameters until the model converges;

[0014] S8: input the cross-domain multi-modal data into the trained model to obtain the output result.

[0015] Further, the input dimensions of the two encoders correspond to the dimensions of the two modal data respectively, and the output dimensions of the two decoders correspond to the dimensions of the two modal data respectively.

[0016] Further, the input dimension of the decoder is equal to twice the output dimension of the encoder.

[0017] Further, each sample of the cross-domain multi-modal dataset is composed of features of two modalities.

[0018] Further, the features of the two modalities are 20-dimensional GIST features and 59-dimensional PHOG features, respectively.

[0019] Further, the loss function of the representation in the latent space is specifically as follows:

[0020]

[0021] wherein B represents the number of batches of samples randomly sampled from the dataset, N represents the number of samples in the dataset, sim represents a similarity function, i represents a count parameter of the batches of samples randomly sampled from the dataset, and j represents a count parameter of the samples in the dataset, and represents a representation of data of the i-th sample of the two modalities in the latent space extracted by the encoder network, represents a representation of data of the i-th sample of the two modalities in the latent space extracted by the encoder network, represents a representation of data of the j-th sample of the two modalities in the latent space extracted by the encoder network.

[0022] Further, the calculation of the reconstruction loss function is specifically represented as follows:

[0023]

[0024]

[0025] wherein, represents data of the i-th sample of the j-th modality in the batch of samples randomly sampled from the dataset, represents data of the i-th sample of the j-th modality in the batch of samples randomly sampled from the dataset, represents data of the i-th sample of the j-th modality in the batch of samples randomly sampled from the dataset, represents the number of samples in the batch of samples randomly sampled from the dataset.

[0026] Further, the updating of the network parameters is specifically as follows:

[0027] Gradients are obtained at each network node, and each network node updates network weights according to a learning rate of the gradient of the current node.

[0028] Further, the learning rate is 0.001.

[0029] Further, after step S7, the method further comprises:

[0030] ​​Based on the collected sample data, the training set and the test set are divided in a ratio of 1:1, and the features of the training set and the test set are extracted respectively using the trained model;

[0031] The support vector machine model is trained using the training set;

[0032] For the test set features, a greedy matching algorithm is used for matching, and the matched features are spliced according to the matching relationship to obtain the spliced representation;

[0033] The support vector machine is tested using the spliced representation.

[0034] The beneficial effects of the present application are as follows:

[0035] The present application improves the accuracy of model recognition by using an encoder to obtain the representation of features in the latent space, performing cross-view contrast learning in the latent space, and using the decoder of each modality to reconstruct the fused representation to maintain the original information of the modality. BRIEF DESCRIPTION OF DRAWINGS

[0036] Fig. 1 is a method flow diagram of the present application.

[0037] Fig. 2 is a schematic diagram of the training network structure of the present application.

[0038] Fig. 3 is a schematic diagram of the inference network structure of the present application. Embodiments of the present application

[0039] In the following description of the technical solutions in the embodiments of the present application, it is obvious that the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0040] In the description of the embodiments of the present application, it should be noted that the indicated orientation or positional relationship is based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship commonly used when the product is used, or the orientation or positional relationship commonly understood by those skilled in the art, or the orientation or positional relationship commonly used when the product is used, only for the convenience of describing the present application and simplifying the description, and is not indicative or implied that the indicated device or element must have a specific orientation, structure and operation, therefore it cannot be understood as a limitation on the present application. In addition, the terms "first", "second" are only used for differentiation, and cannot be understood as indicating or implying relative importance.

[0041] In the description of the embodiments of the present application, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set", "connected" should be understood broadly, for example, can be fixedly connected, can also be detachably connected, or integrally connected; can be directly connected, or indirectly connected through an intermediate medium. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0042] Embodiment 1

[0043] Embodiment 1 of the present application discloses a cross-domain multi-modal data automatic association mapping and fusion recognition method, as shown in Figure 1, and the specific step process is as follows:

[0044] S1: initialize two encoder networks and two decoder networks;

[0045] Among them, the input dimension of the two encoders respectively corresponds to the dimension of the two modal data, and the output dimension of the two decoders respectively corresponds to the dimension of the two modal data.

[0046] S2: construct a cross-domain multi-modal data set, and randomly sample a batch of samples from the data set;

[0047] The multi-modal data refers to the embedding of images, texts and the like;

[0048] In this embodiment, the cross-domain multi-modal data set has 15 categories and 4485 samples, each sample is composed of two modal features, in order to simulate the misalignment phenomenon, 50% of the data set is randomly misaligned;

[0049] The two modal features are 20-dimensional GIST features and 59-dimensional PHOG features respectively.

[0050] Specifically, the cross-domain multi-modal data set is represented as , wherein N is the total number of samples, and represent two modalities, represents the data of the i-th sample of the i-th modality in the batch of samples randomly sampled from the data set.

[0051] As a preferred embodiment, 1024 samples (each sample containing two modal features) are randomly sampled from the data set as a batch of samples, denoted as .

[0052] In this embodiment, the input dimension of the decoder is equal to twice the output dimension of the encoder;

[0053] ​​Specifically, the input dimensions of the two encoders are 20 and 59 respectively, the output dimensions of the two decoders are 20 and 59 respectively, and the input dimensions of the decoders are 40 and 118 respectively.

[0054] S3: For each modality, for each feature in it, extract the representation of all samples in the latent space by the encoder network , as shown in Figure 3.

[0055] S4: Calculate the loss function of the representation in the latent space by the contrastive learning loss function, as follows:

[0056]

[0057] Where B represents the number of samples randomly sampled from the dataset, N represents the number of samples in the dataset, sim represents the similarity function, i represents the count parameter of the batch of samples randomly sampled from the dataset, and j represents the count parameter of the sample in the dataset, and represent the representation of the data of the i-th sample of the two modalities in the latent space extracted by the encoder network, represents the representation of the data of the j-th sample of the i-th modality in the latent space extracted by the encoder network.

[0058] represents the similarity function, , and t is a parameter.

[0059] S5: Concatenate the representation in the latent space, that is, , and decode and reconstruct by the decoder of each modality to obtain .

[0060] S6: Calculate the reconstruction loss function by the decoded sample and the input sample, as follows:

[0061]

[0062]

[0063] Where, represents the data of the i-th sample of the j-th modality in the batch of samples randomly sampled from the dataset, represents the data of the i-th sample of the j-th modality in the batch of samples randomly sampled from the dataset, represents the data of the i-th sample of the j-th modality in the batch of samples randomly sampled from the dataset, represents the data of the i-th sample of the j-th modality in the batch of samples randomly sampled from the dataset, represents the data of the i-th sample of the j-th modality in the batch of samples randomly sampled from the dataset, ​​Indicates the number of samples in the batch of samples randomly sampled from the data set.

[0064] S7: The reconstructed loss function is summed and back-propagated to update the network parameters until the model converges, as shown in FIG. 2.

[0065] The network parameters are updated as follows:

[0066] The gradient is obtained at each network node, and each network node updates the network weight according to the learning rate of the current node gradient;

[0067] Specifically, the learning rate is 0.001.

[0068] According to the collected samples, the training set and the test set are divided in a ratio of 1:1, and the features of the training set and the test set are extracted using the trained model respectively;

[0069] The support vector machine model is trained using the training set features; the support vector machine is a supervised classification model, which realizes classification by finding the optimal boundary between different categories in the data set;

[0070] For the test set features, a greedy matching algorithm is used for matching, that is, the representation of the latent space of the modal is re-paired with the representation of the latent space of the modal After matching, the features are spliced to obtain the spliced representation.

[0071] The spliced representation is used to test the support vector machine.

[0072] S8: Input the cross-domain multi-modal data into the trained model to obtain the output result.

[0073] The present application is not limited to the foregoing specific embodiments. The present application extends to any new feature or any new combination disclosed in the specification, as well as any new method or process steps disclosed or any new combination.

Claims

1. A method for automatic cross-domain multi-modal data association mapping and fusion recognition, characterized in that, The method comprises the following steps: S1: initializing two encoder networks and two decoder networks; S2: constructing a cross-domain multi-modal dataset and randomly sampling a batch of samples from the dataset; S3: extracting representations of all samples in a latent space through the encoder networks; S4: calculating a loss function of the representations in the latent space through a contrastive learning loss function; S5: concatenating the representations in the latent space and decoding them using the two decoder networks respectively; S6: calculating a reconstruction loss function through the decoded samples and the input samples; S7: updating network parameters by backpropagating the summed reconstruction loss function until the model converges; S8: inputting the cross-domain multi-modal data into the trained model to obtain an output result. 2.The cross-domain multi-modal data automatic correlation mapping and fusion recognition method of claim 1, wherein, The input dimensions of the two encoders correspond to the dimensions of the two modal data respectively, and the output dimensions of the two decoders correspond to the dimensions of the two modal data respectively. 3.The cross-domain multi-modal data automatic correlation mapping and fusion recognition method of claim 2, characterized in that, The input dimension of the decoder is equal to twice the output dimension of the encoder. 4.The method of claim 3, wherein, Each sample of the cross-domain multi-modal dataset is composed of features of two modalities. 5.The cross-domain multi-modal data automatic correlation mapping and fusion recognition method of claim 4, characterized in that, The features of the two modalities are 20-dimensional GIST features and 59-dimensional PHOG features respectively. 6.The cross-domain multi-modal data automatic correlation mapping and fusion recognition method of claim 1, wherein, The loss function of the representations in the latent space is as follows: wherein B represents a number of batches of samples randomly sampled from the data set, N represents a number of samples in the data set, sim represents a similarity function, i represents a count parameter of a batch of samples randomly sampled from the data set, and j represents a count parameter of a sample in the data set, and denotes the representation of the data of the i-th sample of the two modalities in the latent space extracted by the encoder network, representing extraction by an encoder network is the representation of the jth sample of the data in the latent space. 7.The cross-domain multi-modal data automatic correlation mapping and fusion recognition method of claim 1, wherein, The calculation of the reconstruction loss function is specifically represented as follows: wherein represents the first batch of samples randomly sampled from the dataset modal first data of one sample, representation modalities corresponding decoder decoded data, is the number of samples in the batch of samples randomly sampled from the dataset. 8.The method of claim 1, wherein, The updating of the network parameters is specifically as follows: obtain the gradient at each network node, and each network node updates the network weight according to the learning rate of the current node gradient. 9.The cross-domain multi-modal data automatic correlation mapping and fusion recognition method of claim 8, wherein, The learning rate is 0.

001.

10. The method of claim 1-9, wherein, After step S7, the following steps are further included: Based on the collected sample data, divide the training set and the test set in a 1:1 ratio, and use the trained model to extract the features of the training set and the test set respectively; Train a support vector machine model using the training set; For the test set features, use a greedy matching algorithm for matching, and concatenate the matched features according to the matching relationship to obtain the concatenated representation; Test the support vector machine using the concatenated representation.

Images