Cross-modal alignment model training method, cross-modal data matching method and device, equipment and medium

Abstract

The invention discloses a cross-modal alignment model training method, a cross-modal data matching method, a cross-modal data matching device, cross-modal data matching equipment and a medium, and relates to the technical field of multi-modal data analysis. A cross-modal alignment model is a model which is constructed based on an encoder, a decoder and a discriminator and is used for carrying out data alignment on different modal data. Comprising the following steps: mapping original data of different modes to the same potential space by using an encoder to obtain potential representation; constructing a correlation alignment loss function based on every two potential representations, calculating a Wasserstein distance between every two potential representations, and optimizing the Wasserstein distance by using a discriminator based on an adversarial learning method; reconstructing each potential representation back to an original mode by using a decoder, and constructing a reconstruction loss function; and training a cross-modal alignment model by using a total loss function and an adversarial learning method, wherein the total loss function is constructed based on a correlation alignment loss function, an optimized Wasserstein distance and a reconstruction loss function.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal data analysis, and in particular to a training method for a cross-modal alignment model, a cross-modal data matching method, an apparatus, a device, and a medium. Background Art

[0002] With the development of artificial intelligence, big data, and computer technology, multimodal data analysis has become an important research direction in numerous fields, including computer vision, natural language processing, bioinformatics, autonomous driving, and remote sensing image analysis. Multimodal data refers to data derived from different information sources or sensors. For example, single-cell sequencing technology in the biomedical field can simultaneously obtain gene expression data and protein expression data, as well as images of the sound and mouth shape during speech recognition. However, data from different modalities often suffer from statistical distribution differences, feature scale mismatches, noise interference, and information loss. Achieving efficient and accurate alignment in latent space has become a key technical challenge in the field of multimodal data fusion.

[0003] Canonical Correlation Analysis (CCA) is currently the most commonly used method for cross-modal data alignment. Its basic idea is to maximize the correlation between two modalities in the latent space through linear transformation, thereby improving the complementarity and consistency of multimodal information. In recent years, with the development of machine learning and deep learning, deep CCA (DCCA), kernel CCA (KCCA), and variational CCA (VCCA) have also been gradually applied.

[0004] However, existing methods still have the following limitations: 1. Traditional CCA and its variants mainly optimize the correlation between different modalities, which may lead to the loss of original modal structure information and affect the interpretability and discriminability of the data; 2. CCA and its variants rely on global statistical characteristics and are easily affected by noise, data missing or measurement errors; 3. CCA needs to calculate the global covariance matrix and perform matrix decomposition, resulting in high computational complexity and difficulty in applying to large-scale datasets. At the same time, traditional CCA relies on complete dataset training and is not suitable for small-batch optimization; 4. Existing CCA methods are mainly based on sample-level correlation and fail to ensure consistency in global statistical distribution between modalities. In the case of uneven distribution of modal data, domain offset or large data noise, it may lead to potential representation degradation and affect the generalization ability of the model.

[0005] Therefore, the above technical problems are in urgent need of being solved by those skilled in the art. Summary of the Invention

[0006] In view of this, the purpose of the present invention is to provide a cross-modal alignment model training method, cross-modal data matching method, device, equipment, and medium, which can ensure the global distribution consistency of different modal data in the latent space, avoid information loss problems, and improve model training efficiency. The specific scheme is as follows:

[0007] In a first aspect, the present application discloses a method for training a cross-modal alignment model, wherein the cross-modal alignment model is a model constructed based on an encoder, a decoder, and a discriminator for aligning data of different modalities, and the method comprises:

[0008] Acquire raw data of different modalities, and map each of the raw data to the same latent space using the encoder to obtain a latent representation of each of the raw data in the latent space;

[0009] Constructing a correlation alignment loss function based on each two potential representations, and calculating a Wasserstein distance between each two potential representations, so as to optimize the Wasserstein distance using the discriminator based on an adversarial learning method to obtain an optimized Wasserstein distance;

[0010] Reconstructing each of the latent representations back to the original modality using the decoder to obtain restored data, and constructing a reconstruction loss function based on the restored data and the corresponding original data;

[0011] A total loss function is constructed based on the correlation alignment loss function, the optimized Wasserstein distance and the reconstruction loss function, and the cross-modal alignment model is trained using the total loss function and the adversarial learning method to obtain a trained cross-modal alignment model.

[0012] Optionally, before using the encoder to map each of the original data to the same latent space to obtain a latent representation of each of the original data in the latent space, the method further includes:

[0013] Preprocessing the raw data; wherein the preprocessing process includes:

[0014] Performing standardization processing on the original data using corresponding standardization rules according to the data type;

[0015] Performing data cleaning and noise reduction processing on the raw data;

[0016] Perform dimensionality reduction processing on the original data.

[0017] Optionally, the standardization processing of the original data using corresponding standardization rules according to the data type includes:

[0018] If the original data is image data, adjusting the resolution of the image data to a preset resolution and performing normalization processing on pixel values ​​in the image data;

[0019] If the original data is text data, word segmentation and word vector conversion are performed on the text data.

[0020] Optionally, the dimension of the latent representation in the latent space is smaller than the dimension of the original data.

[0021] Optionally, the expression of the correlation alignment loss function is:

[0022] ;

[0023] Where L1 is the correlation alignment loss function, 、 is the potential representation of the original data of two different modalities in the latent space, x and y are the original data, f x and f y Indicates an encoder.

[0024] Optionally, the training method of the cross-modal alignment model further includes:

[0025] The Wasserstein distance between each two potential representations is calculated based on a first preset calculation formula; wherein the first preset calculation formula is:

[0026] ;

[0027] Where, is the Wasserstein distance; 、 are the potential representations corresponding to the original data of two different modalities in the latent space; and is the distribution corresponding to the two potential representations in the latent space; inf represents the lower bound, and finds the optimal transmission distance The minimum expected cost; For all joint distributions The set of , satisfying the marginal distribution is and ; is the mathematical expectation, indicating that Weighted average of the distances of all point pairs; Indicates calculating the Euclidean distance.

[0028] Optionally, the training method of the cross-modal alignment model further includes:

[0029] Constraining the 1-Lipschitz continuity of the discriminator based on a preset gradient penalty term;

[0030] Accordingly, the optimized Wasserstein distance is calculated based on a second preset calculation formula, wherein the second preset calculation formula is:

[0031] ;

[0032] Where, is the optimized Wasserstein distance; D is the discriminator; express In distribution Under expectations, Denotes the discriminator D to The expected output; express In distribution The mathematical expectation value under Denotes the discriminator D to The mathematical expectation value of GP is the gradient penalty term, GP= ;in, is the weight hyperparameter of the gradient penalty term, Represents the sampling samples in the latent space Perform expectation calculations, is 、 The samples obtained by linear interpolation between For the discriminator D The gradient, Represents the calculation of the two norm. If the gradient is not 1, a penalty is generated and the square of the difference is minimized to 1.

[0033] Optionally, the reconstruction loss function is expressed as:

[0034] ;

[0035] Where L2 is the reconstruction loss, Indicates that the sample x follows the distribution , Indicates that the sample y follows the distribution ; x and y are original data, f x and f y represents the encoder, 、 are the potential representations corresponding to the original data of two different modalities in the latent space; and For the decoder, and For the recovered data; represents the square of the Euclidean distance, It is the mathematical expectation of all samples of mode 𝑥.

[0036] Optionally, constructing a total loss function based on the correlation alignment loss function, the optimized Wasserstein distance, and the reconstruction loss function includes:

[0037] Performing weighted processing on the correlation alignment loss function, the optimized Wasserstein distance, and the reconstruction loss function based on a preset weight coefficient to construct a total loss function;

[0038] Among them, the total loss function is:

[0039] L=

[0040] ;

[0041] Among them, L is the total loss function; is the encoder f x and f y Parameters; For decoder and Parameters; Represents the joint distribution The following sample pairs Find the mathematical expectation; 、 、 is the weight coefficient.

[0042] Optionally, the process of training the cross-modal alignment model using the total loss function and the adversarial learning method further includes:

[0043] Alternately iterate the first training process and the second training process until the distributions are aligned;

[0044] The first training process includes: fixing the encoder and the decoder, and training the discriminator to maximize the optimized Wasserstein distance;

[0045] The second training process is: fixing the discriminator, and training the encoder and the decoder to minimize the optimized Wasserstein distance.

[0046] In a second aspect, the present application discloses a cross-modal data matching method, comprising:

[0047] Acquire first modal data and second modal data to be matched;

[0048] Inputting the first modality data and the second modality data into a cross-modality alignment model trained by the aforementioned cross-modality alignment model training method to output corresponding first latent representations and second latent representations, respectively;

[0049] calculating a similarity between the first latent representation and the second latent representation;

[0050] If the similarity is greater than a preset threshold, it is determined that the first modal data and the second modal data match, otherwise they do not match.

[0051] In a third aspect, the present application discloses a training device for a cross-modal alignment model, wherein the cross-modal alignment model is a model for aligning data of different modalities based on an encoder, a decoder, and a discriminator, and the device comprises:

[0052] a mapping module, configured to obtain raw data of different modalities and map each of the raw data to a same latent space using the encoder to obtain a latent representation of each of the raw data in the latent space;

[0053] a distance calculation module, configured to construct a correlation alignment loss function based on each two potential representations, and calculate a Wasserstein distance between each two potential representations, so as to optimize the Wasserstein distance using the discriminator based on an adversarial learning method to obtain an optimized Wasserstein distance;

[0054] a data reconstruction module, configured to reconstruct each of the latent representations back to the original modality using the decoder to obtain restored data, and construct a reconstruction loss function based on the restored data and the corresponding original data;

[0055] A training module is used to construct a total loss function based on the correlation alignment loss function, the optimized Wasserstein distance and the reconstruction loss function, and to train the cross-modal alignment model using the total loss function and an adversarial learning method to obtain a trained cross-modal alignment model.

[0056] In a fourth aspect, the present application discloses an electronic device, comprising:

[0057] Memory, used to store computer programs;

[0058] A processor is used to execute the computer program to implement the steps of the aforementioned disclosed cross-modal alignment model training method and cross-modal data matching method.

[0059] In a fifth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the aforementioned disclosed cross-modal alignment model training method and cross-modal data matching method.

[0060] It can be seen that the cross-modal alignment model in this application is a model for aligning data of different modalities based on an encoder, a decoder and a discriminator. When training the cross-modal alignment model, it is necessary to obtain original data of different modalities, and use the encoder to map each of the original data to the same latent space to obtain the potential representation of each of the original data in the latent space; construct a correlation alignment loss function based on each two of the latent representations, and calculate the Wasserstein distance between each two of the latent representations, and optimize the Wasserstein distance using the discriminator based on the adversarial learning method to obtain the optimized Wasserstein distance; use the decoder to reconstruct each of the latent representations back to the original modality to obtain restored data, and construct a reconstruction loss function based on the restored data and the corresponding original data; construct a total loss function based on the correlation alignment loss function, the optimized Wasserstein distance and the reconstruction loss function, and use the total loss function and the adversarial learning method to train the cross-modal alignment model to obtain a trained cross-modal alignment model.

[0061] Beneficial effects: This application needs to use an encoder to map the original data of different modalities to the same latent space to obtain the potential representation of each original data in the latent space, so as to learn the shared potential representation of different modal data, so that different modal data can be aligned in the same latent space. Furthermore, this application needs to calculate the Wasserstein distance between each two potential representations to measure the global similarity of the distribution of different modal data, and optimize the Wasserstein distance using a discriminator based on the adversarial learning method. By introducing the Wasserstein distance and further optimizing the Wasserstein distance through adversarial learning and the discriminator, the global distribution of different modal data in the latent space is ensured to be consistent, which makes up for the lack of global statistical alignment in traditional methods. In addition, this application also uses a decoder to reconstruct the original data and construct a reconstruction loss function to retain the structural information of the original modality, ensuring that the potential representation can completely restore the original data, avoiding the information loss caused by the traditional method due to the simple pursuit of alignment. Finally, the total loss function in this application is constructed based on the correlation alignment loss function, the optimized Wasserstein distance and the reconstruction loss function. It adopts a distribution alignment optimization strategy and avoids the bottleneck of large-scale matrix calculation by jointly optimizing the Wasserstein distance and sample similarity objectives. While maintaining the accuracy of the model, the computational overhead is significantly reduced, thereby improving the training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0063] Figure 1 This is a flow chart of a training method for a cross-modal alignment model disclosed in this application;

[0064] Figure 2 This is a training flowchart of a specific cross-modal alignment model disclosed in this application;

[0065] Figure 3 This is a flow chart of a cross-modal data matching method disclosed in this application;

[0066] Figure 4 This is a schematic diagram of the structure of a training device for a cross-modal alignment model disclosed in this application;

[0067] Figure 5 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0068] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0069] Canonical correlation analysis (CCA) is currently the most commonly used method for cross-modal data alignment. Its basic idea is to maximize the correlation between two modalities in the latent space through linear transformation, thereby improving the complementarity and consistency of multimodal information. In recent years, with the development of machine learning and deep learning, deep CCA, kernel CCA, and variational CCA have also been gradually applied.

[0070] However, existing methods still have the following limitations: 1. Traditional CCA and its variants mainly optimize the correlation between different modalities, which may lead to the loss of original modal structure information and affect the interpretability and discriminability of the data; 2. CCA and its variants rely on global statistical characteristics and are easily affected by noise, data missing or measurement errors; 3. CCA needs to calculate the global covariance matrix and perform matrix decomposition, resulting in high computational complexity and difficulty in applying to large-scale datasets. At the same time, traditional CCA relies on complete dataset training and is not suitable for small-batch optimization; 4. Existing CCA methods are mainly based on sample-level correlation and fail to ensure consistency in global statistical distribution between modalities. In the case of uneven distribution of modal data, domain offset or large data noise, it may lead to potential representation degradation and affect the generalization ability of the model.

[0071] To this end, the embodiments of the present application disclose a training method for a cross-modal alignment model, a cross-modal data matching method, an apparatus, a device and a medium, which can ensure the global distribution consistency of different modal data in the latent space, avoid information loss problems, and improve model training efficiency.

[0072] See also Figure 1 and Figure 2 As shown, an embodiment of the present application discloses a training method for a cross-modal alignment model, wherein the cross-modal alignment model is a model for aligning data of different modalities based on an encoder, a decoder, and a discriminator. The method includes:

[0073] Step S11: acquiring original data of different modalities, and mapping each of the original data to the same latent space using the encoder to obtain a latent representation of each of the original data in the latent space.

[0074] In this embodiment, after obtaining the original data of different modalities, since the distribution, latitude, scale and representation method of the data of different modalities are different, it is necessary to use an encoder to map the original data of different modalities to the same latent space to obtain the latent representation of each original data in the latent space, thereby learning the shared latent representation of the data of different modalities, so that the data of different modalities can be aligned in the same latent space.

[0075] Among them, before using the encoder to map each of the original data to the same latent space to obtain the latent representation of each of the original data in the latent space, it also includes: preprocessing the original data; wherein the preprocessing process includes: standardizing the original data using corresponding standardization rules according to the data type; performing data cleaning and noise reduction on the original data; and performing dimensionality reduction on the original data.

[0076] That is, in order to ensure that cross-modal data can be processed under the same framework, the embodiment of the present application also needs to pre-process the input raw data to ensure that the cross-modal data has a unified format, high-quality signals and appropriate alignment when input, providing a solid foundation for subsequent feature extraction and alignment optimization. In a specific embodiment, the pre-processing process may specifically include but is not limited to standardizing the raw data using corresponding standardization rules according to the data type, performing data cleaning and noise reduction on the raw data, performing dimensionality reduction on the raw data, and so on.

[0077] It is understandable that this embodiment can obtain data sets of multiple modalities, such as text, images, audio, biological signals, etc., so this application needs to standardize the data of different modalities to eliminate the problem of data scale mismatch. In a specific embodiment, the raw data is standardized according to the data type using corresponding standardization rules, including: if the raw data is image data, the resolution of the image data is adjusted to a preset resolution, and the pixel values ​​in the image data are normalized; if the raw data is text data, the text data is segmented and converted into word vectors. That is, this embodiment can use corresponding standardization rules to perform standardization according to the data type. For image data, the resolution of the image data can be unified, such as adjusting it to a preset resolution, and the pixel values ​​can also be normalized, such as scaling to the (0,1) interval. The uniformity of the input image is ensured by adjusting the image resolution, color and other information. For text data, the text data can be segmented and converted into word vectors to convert them into numerical features.

[0078] Furthermore, since cross-modal data typically comes from different sensors or data sources, it contains noise, missing values, and outliers. Therefore, this embodiment improves data quality through data cleaning and denoising operations. For example, Gaussian filtering or median filtering can be used to remove noise from image data; stop words and irrelevant symbols can be removed from text data; and missing values ​​can be interpolated or masked. Furthermore, since the data dimensions of different modalities may vary significantly, this embodiment can also perform feature extraction and dimensionality reduction during data preprocessing to improve computational efficiency and reduce computational complexity. For example, dimensionality reduction can be performed using principal component analysis (PCA) or a deep autoencoder.

[0079] Taking the original data of two modalities as an example, after data preprocessing, the preprocessed original data is obtained and , where p and q represent data dimensions, and R represents a set of real numbers. Furthermore, in order to map data of different modalities into a shared latent space for cross-modal data alignment, this application constructs two cross-modal mapping functions (i.e., encoders): and , Responsible for the modal Convert the data (such as pictures) into k-dimensional latent representation , Responsible for the modal Convert the data (such as pictures) into k-dimensional latent representation , k is the dimension of the latent space. That is, this application will and As a nonlinear mapping network, construct mode x and mode y and Achieve the mapping of the shared latent space and obtain and The potential representation of and This is achieved through deep neural networks.

[0080] It should be noted that the dimension of the latent representation in the latent space is smaller than the dimension of the original data. In other words, k is usually much smaller than the dimensions p and q of the original data to reduce computational complexity and improve feature extraction capabilities.

[0081] Step S12: constructing a correlation alignment loss function based on each two of the potential representations, and calculating the Wasserstein distance between each two of the potential representations, so as to optimize the Wasserstein distance using the discriminator based on the adversarial learning method to obtain an optimized Wasserstein distance.

[0082] In this embodiment, the focus is on the alignment between data of two different modalities. If there are more than two modalities, the data of each two different modalities are aligned separately.

[0083] Therefore, after obtaining the latent representation of each raw data in the latent space, it is necessary to construct a correlation alignment loss function based on each pair of latent representations to maximize the canonical correlation between the two modal latent representations. The Wasserstein distance between each pair of latent representations is then calculated to measure the global similarity of the distribution of data from different modalities. The Wasserstein distance is then optimized using a discriminator based on adversarial learning to obtain the optimized Wasserstein distance. By introducing the Wasserstein distance and further optimizing it through adversarial learning and the discriminator, the global distribution of data from different modalities in the latent space is ensured to be consistent, thus compensating for the lack of global statistical alignment in traditional methods.

[0084] Among them, the expression of the correlation alignment loss function is:

[0085] ;

[0086] Where L1 is the correlation alignment loss function, 、 is the potential representation of the original data of two different modalities in the latent space, x and y are the original data, f x and f y Indicates an encoder.

[0087] Furthermore, the above method further includes: calculating the Wasserstein distance between each two potential representations based on a first preset calculation formula; wherein the first preset calculation formula is:

[0088] ;

[0089] Where, is the Wasserstein distance; 、 are the potential representations corresponding to the original data of two different modalities in the latent space; and is the distribution corresponding to the two potential representations in the latent space; inf represents the lower bound, and finds the optimal transmission distance The minimum expected cost; For all joint distributions The set of , satisfying the marginal distribution is and ; is the mathematical expectation, indicating that Weighted average of the distances of all point pairs; Indicates calculating the Euclidean distance.

[0090] It is understandable that in traditional CCA methods, the main focus is on sample-level correlation, that is, making the same set of data (such as a picture and its text description) close in the latent space. However, this method ignores the global statistical distribution of the data and is easily affected by abnormal samples. The embodiment of the present application introduces the Wasserstein distance for global distribution alignment. Unlike the traditional KL divergence (Kullback-Leibler Divergence) or JS divergence (Jensen-Shannon Divergence), the Wasserstein distance not only considers the point-to-point distance between distributions, but also focuses on the optimal transmission method of data points between distributions. Therefore, it is more stable and interpretable. Wasserstein distance is widely used in many machine learning tasks, especially in the fields of generative adversarial networks, probability density estimation, domain adaptation and cross-modal data alignment, and can effectively measure the similarity between different distributions. Ensure that the potential representations of different modalities are not only locally related, but also consistent in global statistical distribution: It measures the distance between the potential distributions of different modalities. Minimizing this distance means that the potential representations of the two modalities are consistent at the distribution level, that is, achieving global distribution alignment of different modal data in the shared latent space.

[0091] Furthermore, the above method further includes: constraining the 1-Lipschitz continuity of the discriminator based on a preset gradient penalty term;

[0092] Accordingly, the optimized Wasserstein distance is calculated based on a second preset calculation formula, wherein the second preset calculation formula is:

[0093] ;

[0094] Where, is the optimized Wasserstein distance; D is the discriminator; express In distribution Under expectations, Denotes the discriminator D to The expected output; express In distribution The mathematical expectation value under Denotes the discriminator D to The mathematical expectation value of GP is the gradient penalty term, GP= ;in, is the weight hyperparameter of the gradient penalty term, Represents the sampling samples in the latent space Perform expectation calculations, is 、 The samples obtained by linear interpolation between For the discriminator D The gradient, Represents the calculation of the two norm. If the gradient is not 1, a penalty is generated and the square of the difference is minimized to 1.

[0095] It is understandable that this application adopts an adversarial learning strategy and designs a discriminator D that satisfies the 1-Lipschitz condition to optimize the Wasserstein distance between potential representations by maximizing , and combined with the gradient penalty term GP to ensure the stability of the Wasserstein distance estimate. That is, this application adopts an adversarial learning strategy to optimize the Wasserstein distance estimate, and combines the gradient penalty term to ensure the 1-Lipschitz constraint of the discriminator.

[0096] As can be seen, this application uses a latent representation learning method based on autoencoders, combined with a Wasserstein distance optimization strategy, to ensure that data from different modalities have similar distributions in a shared latent space. By using autoencoders to enhance the ability to retain modal information, and combining adversarial learning to optimize the Wasserstein distance estimation, the stability and generalization of cross-modal data alignment are improved.

[0097] Step S13: Utilize the decoder to reconstruct each of the latent representations back to the original modality to obtain restored data, and construct a reconstruction loss function based on the restored data and the corresponding original data.

[0098] In this embodiment, the decoder is used to reconstruct the original data and construct a reconstruction loss function to preserve the structural information of the original modality, ensuring that the potential representation can completely restore the original data, avoiding the information loss caused by the traditional method of simply pursuing alignment.

[0099] That is, in order to preserve the original information structure of each modality, the embodiment of the present application also constructs a decoder: and ,and and With the aforementioned encoder and Symmetric, together they constitute the framework of the adversarial autoencoder. Among them, by taking advantage of the ability to retain the modal information of the autoencoder, the decoder and Reconstruct each latent representation back to the original modality to obtain the recovered data, that is, the decoder is used to reconstruct the original data from the latent representation, i.e. and .

[0100] Among them, the expression of the reconstruction loss function is:

[0101] ;

[0102] Where L2 is the reconstruction loss, Indicates that the sample x follows the distribution , Indicates that the sample y follows the distribution ; x and y are original data, f x and f y represents the encoder, 、 are the potential representations corresponding to the original data of two different modalities in the latent space; and For the decoder, and For the recovered data; represents the square of the Euclidean distance, It is the mathematical expectation of all samples of mode 𝑥.

[0103] It can be understood that the embodiments of the present application minimize the reconstruction loss to ensure that the potential representation can completely restore the original data, thereby preserving the original modal information and reducing information loss.

[0104] Step S14: constructing a total loss function based on the correlation alignment loss function, the optimized Wasserstein distance and the reconstruction loss function, and using the total loss function and the adversarial learning method to train the cross-modal alignment model to obtain a trained cross-modal alignment model.

[0105] In this embodiment, the total loss function is constructed based on the correlation alignment loss function, the optimized Wasserstein distance and the reconstruction loss function, and adopts a distribution alignment optimization strategy. By jointly optimizing the Wasserstein distance and sample similarity objectives, the bottleneck of large-scale matrix calculation is avoided, so that the computational overhead of the model is significantly reduced while maintaining accuracy, thereby improving training efficiency.

[0106] In a specific embodiment, constructing a total loss function based on the correlation alignment loss function, the optimized Wasserstein distance and the reconstruction loss function includes: weighting the correlation alignment loss function, the optimized Wasserstein distance and the reconstruction loss function based on a preset weight coefficient to construct a total loss function;

[0107] Among them, the total loss function is:

[0108] L=

[0109] ;

[0110] Among them, L is the total loss function; is the encoder f x and f y Parameters; For decoder and Parameters; Represents the joint distribution The following sample pairs Find the mathematical expectation; 、 、 is the weight coefficient. Controls the weight of the optimized Wasserstein distance, 、 Control the modal feature preservation ability to balance the optimization objectives of correlation alignment, distribution alignment and reconstruction loss.

[0111] By optimizing the above total loss function, we can learn the shared latent representation of data from different modalities and achieve effective alignment of cross-modal data. In practical applications, this method can be applied to multimodal data analysis tasks in fields such as computer vision, natural language processing, and bioinformatics.

[0112] Furthermore, the above-mentioned process of training the cross-modal alignment model using the total loss function and the adversarial learning method also includes: alternating the first training process and the second training process until the distribution is aligned; wherein, the first training process is: fixing the encoder and the decoder, and training the discriminator to maximize the optimized Wasserstein distance; the second training process is: fixing the discriminator, and training the encoder and the decoder to minimize the optimized Wasserstein distance.

[0113] That is, the present application adopts an alternating training strategy during the training process, with the first and second training processes iterating alternately until the distributions align (i.e., converge). In the first training process, the encoder and decoder are fixed, and the discriminator D is trained to maximize the Wasserstein distance estimate; in the second training process, the discriminator is fixed, and the encoder and decoder are trained to minimize the optimized Wasserstein distance, that is, to minimize the total loss function.

[0114] This application proposes a cross-modal data alignment method based on Wasserstein Canonical Correlation Analysis (Wasserstein CCA, wCCA). Through deep neural network mapping, data from different modalities are converted into a shared latent representation. Global distribution alignment is achieved by minimizing the Wasserstein distance, ensuring that the data from different modalities have consistent statistical properties in the latent space. This application utilizes an autoencoder for feature reconstruction to maintain the information integrity of the original modality and avoid information loss during the data alignment process. Furthermore, adversarial learning is used to further optimize the calculation of the Wasserstein distance, enabling this method to achieve efficient and robust cross-modal alignment in tasks such as computer vision, natural language processing, bioinformatics, and remote sensing image analysis. Furthermore, this application maintains stable optimization even with small-batch training, making it suitable for large-scale datasets. It can be widely applied to cross-modal retrieval, matching, classification, and data augmentation tasks, improving the fusion and alignment of cross-modal data.

[0115] This solution has the following technical effects:

[0116] (1) Strong cross-modal alignment capability and maintaining information integrity: This application is based on the Wasserstein distance constraint, which ensures the distribution consistency of data in the shared latent space. It can improve the cross-modal alignment capability while retaining category information to the greatest extent, thus avoiding the problem of information loss.

[0117] (2) Robust and applicable to a variety of data types: This application is based on an adversarial autoencoder structure, which has strong adaptability to data noise, missing values, and different distribution forms. In practical applications, it shows good stability on multimodal data and is applicable to a variety of complex practical application scenarios.

[0118] (3) Optimize the calculation process and improve training efficiency: Traditional CCA methods require large-scale data calculations and are difficult to adapt to the small-batch training model of deep learning. This application adopts a distribution alignment optimization strategy. By jointly optimizing the Wasserstein distance and sample similarity objectives, it avoids the bottleneck of large-scale matrix calculations. While maintaining the accuracy of the model, the computational overhead is significantly reduced, thereby improving training efficiency.

[0119] It can be seen that the present application needs to use an encoder to map the original data of different modalities to the same latent space to obtain the potential representation of each original data in the latent space, so as to learn the shared potential representation of different modal data so that different modal data can be aligned in the same latent space. Furthermore, the present application needs to calculate the Wasserstein distance between each two potential representations to measure the global similarity of the distribution of different modal data, and optimize the Wasserstein distance using a discriminator based on the adversarial learning method. By introducing the Wasserstein distance and further optimizing the Wasserstein distance through adversarial learning and the discriminator, the global distribution of different modal data in the latent space is ensured to be consistent, which makes up for the lack of global statistical alignment in traditional methods. In addition, the present application also uses a decoder to reconstruct the original data and construct a reconstruction loss function to retain the structural information of the original modality, ensuring that the potential representation can completely restore the original data, avoiding the information loss caused by the traditional method due to the simple pursuit of alignment. Finally, the total loss function in this application is constructed based on the correlation alignment loss function, the optimized Wasserstein distance and the reconstruction loss function. It adopts a distribution alignment optimization strategy and avoids the bottleneck of large-scale matrix calculation by jointly optimizing the Wasserstein distance and sample similarity objectives. While maintaining the accuracy of the model, the computational overhead is significantly reduced, thereby improving the training efficiency.

[0120] See also Figure 3 As shown, the embodiment of the present application discloses a cross-modal data matching method, including:

[0121] Step S21: Acquire the first modal data and the second modal data to be matched.

[0122] In this embodiment, when cross-modal data matching is required, firstly, the first modal data and the second modal data to be matched are obtained.

[0123] Step S22: input the first modal data and the second modal data into the cross-modal alignment model trained using the aforementioned disclosed cross-modal alignment model training method to output the corresponding first latent representation and second latent representation, respectively.

[0124] In this embodiment, the cross-modal alignment model obtained by the above training can be specifically applied to the cross-modal data matching task, by inputting the first modality data and the second modality data into the cross-modal alignment model to output the corresponding first latent representation and second latent representation respectively.

[0125] Step S23: Calculate the similarity between the first latent representation and the second latent representation.

[0126] In this embodiment, after the latent representation is obtained, the similarity between the two latent representations is calculated. In a specific implementation, the cosine distance can be used to calculate the similarity between the two.

[0127] Step S24: If the similarity is greater than a preset threshold, it is determined that the first modal data and the second modal data match, otherwise they do not match.

[0128] In this embodiment, if the similarity is greater than a preset threshold, the first modal data and the second modal data are determined to match, otherwise they are not matched. That is, the present application matches data of different modalities through the learned shared latent representation, and determines whether the two data match by calculating the similarity score of the cross-modal data. The above method can be widely applied to application scenarios such as image-text matching, speech-text matching, and biomedical data matching.

[0129] In addition to being applicable to cross-modal data matching tasks, cross-modal alignment models can also be applied to cross-modal data retrieval, classification, and data augmentation tasks. The principle is to use a mapping function to transform data from different modalities into a shared latent representation space, and then use Wasserstein Canonical Correlation Analysis (wCCA) to determine whether data from two different modalities have similar semantic information.

[0130] It can be seen that the present application can use the cross-modal alignment model obtained by the above training to be applied in the cross-modal data matching scenario, by inputting the first modal data and the second modal data to be matched into the cross-modal alignment model to output the corresponding first latent representation and the second latent representation respectively, and then determining whether the two data match by calculating the similarity between the two latent representations.

[0131] See also Figure 4 As shown, an embodiment of the present application discloses a training device for a cross-modal alignment model, wherein the cross-modal alignment model is a model for aligning data of different modalities based on an encoder, a decoder, and a discriminator. The device includes:

[0132] a mapping module 11 for acquiring raw data of different modalities and mapping each of the raw data to the same latent space using the encoder to obtain a latent representation of each of the raw data in the latent space;

[0133] a distance calculation module 12, configured to construct a correlation alignment loss function based on each two potential representations, and calculate a Wasserstein distance between each two potential representations, so as to optimize the Wasserstein distance using the discriminator based on an adversarial learning method to obtain an optimized Wasserstein distance;

[0134] a data reconstruction module 13, configured to reconstruct each of the latent representations back to the original modality using the decoder to obtain restored data, and construct a reconstruction loss function based on the restored data and the corresponding original data;

[0135] The training module 14 is used to construct a total loss function based on the correlation alignment loss function, the optimized Wasserstein distance and the reconstruction loss function, and use the total loss function and the adversarial learning method to train the cross-modal alignment model to obtain a trained cross-modal alignment model.

[0136] It can be seen that the present application needs to use an encoder to map the original data of different modalities to the same latent space to obtain the potential representation of each original data in the latent space, so as to learn the shared potential representation of different modal data so that different modal data can be aligned in the same latent space. Furthermore, the present application needs to calculate the Wasserstein distance between each two potential representations to measure the global similarity of the distribution of different modal data, and optimize the Wasserstein distance using a discriminator based on the adversarial learning method. By introducing the Wasserstein distance and further optimizing the Wasserstein distance through adversarial learning and the discriminator, the global distribution of different modal data in the latent space is ensured to be consistent, which makes up for the lack of global statistical alignment in traditional methods. In addition, the present application also uses a decoder to reconstruct the original data and construct a reconstruction loss function to retain the structural information of the original modality, ensuring that the potential representation can fully restore the original data, and avoiding the information loss caused by the traditional method due to the simple pursuit of alignment. Finally, the total loss function in this application is constructed based on the correlation alignment loss function, the optimized Wasserstein distance and the reconstruction loss function. It adopts a distribution alignment optimization strategy and avoids the bottleneck of large-scale matrix calculation by jointly optimizing the Wasserstein distance and sample similarity objectives. While maintaining the accuracy of the model, the computational overhead is significantly reduced, thereby improving the training efficiency.

[0137] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Specifically, it may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps of the cross-modal alignment model training method and cross-modal data matching method performed by the electronic device disclosed in any of the aforementioned embodiments.

[0138] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0139] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0140] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon include an operating system 221, a computer program 222 and data 223, etc. The storage method can be temporary storage or permanent storage.

[0141] Among them, the operating system 221 is used to manage and control the various hardware devices and computer programs 222 on the electronic device 20, so as to realize the calculation and processing of the massive data 223 in the memory 22 by the processor 21, which can be Windows, Unix, Linux, etc. In addition to including computer programs that can be used to complete the training method of the cross-modal alignment model and the cross-modal data matching method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include computer programs that can be used to complete other specific tasks. In addition to including data received by the electronic device and transmitted from an external device, the data 223 can also include data collected by its own input and output interface 25, etc.

[0142] Furthermore, an embodiment of the present application also discloses a computer-readable storage medium, which stores a computer program. When the computer program is loaded and executed by a processor, it implements the training method of the cross-modal alignment model and the cross-modal data matching method steps disclosed in any of the aforementioned embodiments.

[0143] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0144] Those skilled in the art may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the composition and steps of each example according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0145] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a compact disc read-only memory (CD-ROM), or any other form of storage medium known in the art.

[0146] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0147] The above is a detailed introduction to the training method of the cross-modal alignment model, the cross-modal data matching method, the device, the equipment and the medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A training method for a cross-modal alignment model, characterized in that: The cross-modal alignment model is a model built based on an encoder, a decoder, and a discriminator for aligning data of different modalities. The method includes: Acquire raw data of different modalities, and map each of the raw data to the same latent space using the encoder to obtain a latent representation of each of the raw data in the latent space; Constructing a correlation alignment loss function based on each two potential representations, and calculating a Wasserstein distance between each two potential representations, so as to optimize the Wasserstein distance using the discriminator based on an adversarial learning method to obtain an optimized Wasserstein distance; Reconstructing each of the latent representations back to the original modality using the decoder to obtain restored data, and constructing a reconstruction loss function based on the restored data and the corresponding original data; A total loss function is constructed based on the correlation alignment loss function, the optimized Wasserstein distance and the reconstruction loss function, and the cross-modal alignment model is trained using the total loss function and the adversarial learning method to obtain a trained cross-modal alignment model.

2. The method for training a cross-modal alignment model according to claim 1, wherein: Before mapping the original data to the same latent space using the encoder to obtain a latent representation of the original data in the latent space, the method further includes: Preprocessing the raw data; wherein the preprocessing process includes: Performing standardization processing on the original data using corresponding standardization rules according to the data type; Performing data cleaning and noise reduction processing on the raw data; Perform dimensionality reduction processing on the original data.

3. The method for training a cross-modal alignment model according to claim 2, wherein: The standardization processing of the original data using corresponding standardization rules according to the data type includes: If the original data is image data, adjusting the resolution of the image data to a preset resolution and performing normalization processing on pixel values ​​in the image data; If the original data is text data, word segmentation and word vector conversion are performed on the text data.

4. The method for training a cross-modal alignment model according to claim 1, wherein: The dimension of the latent representation in the latent space is smaller than the dimension of the original data.

5. The method for training a cross-modal alignment model according to claim 1, wherein: The expression of the correlation alignment loss function is: ； Where L1 is the correlation alignment loss function, 、 is the potential representation of the original data of two different modalities in the latent space, x and y are the original data, f x and f y Indicates an encoder.

6. The method for training a cross-modal alignment model according to claim 5, wherein: Also includes: The Wasserstein distance between each two potential representations is calculated based on a first preset calculation formula; wherein the first preset calculation formula is: ； Where, is the Wasserstein distance; 、 are the potential representations corresponding to the original data of two different modalities in the latent space; and is the distribution corresponding to the two potential representations in the latent space; inf represents the lower bound, and finds the optimal transmission distance The minimum expected cost; For all joint distributions The set of , satisfying the marginal distribution is and ; is the mathematical expectation, indicating that Weighted average of the distances of all point pairs; Indicates calculating the Euclidean distance.

7. The method for training a cross-modal alignment model according to claim 6, wherein: Also includes: Constraining the 1-Lipschitz continuity of the discriminator based on a preset gradient penalty term; Accordingly, the optimized Wasserstein distance is calculated based on a second preset calculation formula, wherein the second preset calculation formula is: ； Where, is the optimized Wasserstein distance; D is the discriminator; express In distribution Under expectations, Denotes the discriminator D to The expected output; express In distribution The mathematical expectation value under Denotes the discriminator D to The mathematical expectation value of GP is the gradient penalty term, GP= ;in, is the weight hyperparameter of the gradient penalty term, Represents the sampling samples in the latent space Perform expectation calculations, is 、 The samples obtained by linear interpolation between For the discriminator D The gradient, Represents the calculation of the two norm. If the gradient is not 1, a penalty is generated and the square of the difference is minimized to 1.

8. The method for training a cross-modal alignment model according to claim 7, wherein: The expression of the reconstruction loss function is: ； Where L2 is the reconstruction loss, Indicates that the sample x follows the distribution , Indicates that the sample y follows the distribution ; x and y are original data, f x and f y represents the encoder, 、 are the potential representations corresponding to the original data of two different modalities in the latent space; and For the decoder, and For the recovered data; represents the square of the Euclidean distance, It is the mathematical expectation of all samples of mode 𝑥.

9. The method for training a cross-modal alignment model according to claim 8, wherein: The constructing of a total loss function based on the correlation alignment loss function, the optimized Wasserstein distance and the reconstruction loss function includes: Performing weighted processing on the correlation alignment loss function, the optimized Wasserstein distance, and the reconstruction loss function based on a preset weight coefficient to construct a total loss function; Among them, the total loss function is: L= ； Among them, L is the total loss function; is the encoder f x and f y Parameters; For decoder and Parameters; Represents the joint distribution The following sample pairs Find the mathematical expectation; 、 、 is the weight coefficient.

10. The method for training a cross-modal alignment model according to any one of claims 1 to 9, characterized in that: The process of training the cross-modal alignment model using the total loss function and the adversarial learning method further includes: Alternately iterate the first training process and the second training process until the distributions are aligned; The first training process includes: fixing the encoder and the decoder, and training the discriminator to maximize the optimized Wasserstein distance; The second training process is: fixing the discriminator, and training the encoder and the decoder to minimize the optimized Wasserstein distance.

11. A cross-modal data matching method, characterized in that: include: Acquire first modal data and second modal data to be matched; Inputting the first modal data and the second modal data into a cross-modal alignment model trained using the cross-modal alignment model training method according to any one of claims 1 to 10 to output corresponding first latent representations and second latent representations, respectively; calculating a similarity between the first latent representation and the second latent representation; If the similarity is greater than a preset threshold, it is determined that the first modal data and the second modal data match, otherwise they do not match.

12. A training device for a cross-modal alignment model, characterized in that: The cross-modal alignment model is a model for aligning data of different modalities based on an encoder, a decoder, and a discriminator. The device includes: a mapping module, configured to obtain raw data of different modalities and map each of the raw data to a same latent space using the encoder to obtain a latent representation of each of the raw data in the latent space; a distance calculation module, configured to construct a correlation alignment loss function based on each two potential representations, and calculate a Wasserstein distance between each two potential representations, so as to optimize the Wasserstein distance using the discriminator based on an adversarial learning method to obtain an optimized Wasserstein distance; a data reconstruction module, configured to reconstruct each of the latent representations back to the original modality using the decoder to obtain restored data, and construct a reconstruction loss function based on the restored data and the corresponding original data; A training module is used to construct a total loss function based on the correlation alignment loss function, the optimized Wasserstein distance and the reconstruction loss function, and to train the cross-modal alignment model using the total loss function and an adversarial learning method to obtain a trained cross-modal alignment model.

13. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that Used to store a computer program; wherein, when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images