Multimodal variational autoencoder model training method, system and related equipment

Abstract

The present invention is applicable to the field of multimodal data processing and provides a multimodal variational autoencoder model training method, system, and related equipment. The method comprises: obtaining image data containing multiple modalities and pairing each pair of images of different modalities to obtain an image pair dataset; constructing a multimodal variational autoencoder framework model and using the image pair dataset as input to train the multimodal variational autoencoder framework model; using a four-tuple metric loss as the loss function for training the multimodal variational autoencoder framework model until the loss function converges, and then outputting the trained multimodal variational autoencoder model. The present invention achieves alignment of shared latent vectors in the latent space during the training of the multimodal variational autoencoder model, thereby improving the model's ability to represent and generate multimodal data.

Description

Technical Field

[0001] The present invention belongs to the field of multimodal data processing, and in particular relates to a multimodal variational autoencoder model training method, system and related equipment. Background Art

[0002] Multimodal data processing is widely used in fields such as natural science research and engineering technology. Data in different modalities often represent the same thing in different forms, each with unique properties. Modeling and representing multimodal data is a key topic in artificial intelligence research, and multimodal data processing is also a crucial component of AI applications.

[0003] Using probabilistic generative models to process multimodal data is an important research field. Early models could not effectively handle model training and inference problems in large-scale data scenarios. The introduction of Variational Auto-Encoder (VAE) has solved this problem well. Research on deep probabilistic generative models based on the VAE framework has also become an important direction in current multimodal data processing. After 2016, some research work based on the VAE framework has achieved data crosstalk and transformation between modalities by modeling the conditional generative process of data. However, these works did not model multimodal data from a generative perspective, which limited the model's representation capabilities. After 2017, research on modeling the joint distribution of multimodal data has been launched, including the joint multimodal variational auto-encoder (JMVAE) model, the multimodal variational auto-encoder (MVAE) model, and the mixture-of-experts multimodal variational auto-encoder (MMVAE) model. These models achieve representation and conditional generation of multimodal data by modeling the joint probability distribution of multimodal data. Compared with modeling the conditional probability distributions in different directions separately, these models have lower training costs and more effective data representation. However, these models do not decouple the data representation.

[0004] Some research works since 2020 have attempted to represent the shared and private information of different modalities separately. Related work proposed a disentangling multimodal variational auto-encoder (DMVAE) model. This model represents modal sharing and private information separately, and decouples the representation by minimizing the mutual information between shared and private latent vectors. At the same time, it aligns the shared latent vector in the latent space through noise adversarial estimation loss, achieving good results. However, the model still has problems with unclear data generation quality and inaccurate extraction of shared private information.

[0005] In the current field of multimodal data processing, existing research work has no explicit constraints on the shared information between different modal data, which makes it impossible to efficiently decouple the shared and private information of multimodal data, leading to inaccurate information extraction and ambiguous generated data. Summary of the Invention

[0006] To solve the above problems, embodiments of the present invention provide a multimodal variational autoencoder model training method, system, and related equipment, aiming to improve the data generation and decoupling capabilities of shared and private information in multimodal data.

[0007] In a first aspect, an embodiment of the present invention provides a multimodal variational autoencoder model training method, the method comprising the following steps:

[0008] Acquire image data containing multiple modalities and pair every two images of different modalities to obtain an image pair dataset;

[0009] Constructing a multimodal variational autoencoder framework model, and using the image pair dataset as input of the multimodal variational autoencoder framework model to train the multimodal variational autoencoder framework model;

[0010] The quadruple metric loss is used as the loss function for training the multimodal variational autoencoder framework model until the loss function converges, and the trained multimodal variational autoencoder model is output.

[0011] Furthermore, one of the pairs of images in the image pair dataset is defined as a first modality image x and a second modality image y, and a pair of two-modality data obtained by pairing the first modality image x and the second modality image y is (x, y), and the two-modality data (x, y) has a shared latent vector z and private latent vectors h corresponding to the first modality image x and the second modality image y respectively. x 、h y ;

[0012] The multimodal variational autoencoder framework model includes a first generator corresponding to the first modality image x and the second modality image y respectively. Second generator The multimodal variational autoencoder framework model also includes the private latent vectors h x and the private latent vector h y The first private information encoder Second private information encoder and first shared information encoders corresponding to the first modality image x and the second modality image y, respectively First shared information encoder

[0013] Furthermore, the multimodal variational autoencoder framework model models the joint probability distribution of the two modal data (x, y) to satisfy the following relationship (1):

[0014]

[0015] In the relation (1), p(z), p(h x )、p(h y ) are the shared latent vector z, the private latent vector h x , the private latent vector h y The prior distribution of , and they all obey the isotropic Gaussian distribution.

[0016] Furthermore, when the multimodal variational autoencoder framework model is trained, for the two-modal data (x, y), its marginal probability distribution p θ (x,y) satisfies the following relationship (2):

[0017]

[0018] The multimodal variational autoencoder framework model uses variational inference method, using q(z,h x ,h y |x,y) as a variational distribution and calculate lnp θ Variational lower bound on (x,y) The variational lower bound The following relationship (3) is satisfied:

[0019]

[0020] Furthermore, when the multimodal variational autoencoder framework model is trained, for the case where the two modal data (x, y) have missing modal data, its marginal probability distribution p θ (x), p θ(y) satisfies the following relations (4) and (5):

[0021]

[0022]

[0023] The multimodal variational autoencoder framework model uses variational inference method, using q(z,h x ,|x)、q(z,h y ,|y) as the variational distribution and calculate lnp θ The variational lower bound of (x) or lnp θ The variational lower bound of (y) Variational lower bound The following relationship (6) is satisfied:

[0024]

[0025] Variational lower bound The following relationship (7) is satisfied:

[0026]

[0027] Furthermore, the negative samples corresponding to the first modality image x and the second modality image y are defined as x_ and y_, respectively, and the constraints of the four-tuple metric loss are The following relationship (8) is satisfied:

[0028]

[0029] In equation (8), μ x (x) and μ y (y) respectively represent the first shared information encoder The second shared information encoder The mean of the output results, α1 and α2 are hyper parameters;

[0030] The final objective function of the multimodal variational autoencoder framework model is The following relationship (9) is satisfied:

[0031]

[0032] Among them, β is a hyperparameter.

[0033] Furthermore, the multimodal variational autoencoder model is optimized using a gradient descent method during training, using Adam as the optimizer.

[0034] In a second aspect, an embodiment of the present invention further provides a multimodal variational autoencoder model training system, comprising:

[0035] The data acquisition module is used to acquire image data containing multiple modalities and pair every two images of different modalities to obtain an image pair dataset;

[0036] A model construction module is used to construct a multimodal variational autoencoder framework model, and use the image pair dataset as input of the multimodal variational autoencoder framework model to train the multimodal variational autoencoder framework model;

[0037] An optimization module is used to use the quadruple metric loss as the loss function for training the multimodal variational autoencoder framework model until the loss function converges, and output the trained multimodal variational autoencoder model.

[0038] In a third aspect, an embodiment of the present invention further provides a computer device comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the computer program, the steps in the multimodal variational autoencoder model training method as described in any one of the above embodiments are implemented.

[0039] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the multimodal variational autoencoder model training method as described in any one of the above embodiments are implemented.

[0040] The beneficial effect achieved by the present invention lies in proposing a quadruplet metric loss based multimodal variational auto-encoder model (Q-MVAE) and its training method. During the training process, shared and private information are modeled separately, and the joint probability distribution of different modalities and the unimodal marginal probability distribution are modeled. At the same time, a loss function is used to constrain the alignment of shared latent vectors in the latent space, thereby improving the data generation and representation capabilities of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a flowchart of the steps of the multimodal variational autoencoder model training method provided by an embodiment of the present invention;

[0042] Figure 2 Schematic diagram of the logical structure of the multimodal variational autoencoder framework model provided by an embodiment of the present invention;

[0043] Figure 3 is an example of an image sample provided by an embodiment of the present invention;

[0044] Figure 4 This is the effect diagram of the qualitative experimental verification of the Q-MVAE model and the existing model in the data cross-generation experiment;

[0045] Figure 5 This is the effect diagram of the qualitative experimental verification of the Q-MVAE model and the existing DMVAE model in the data conversion generation experiment;

[0046] Figure 6 This is a diagram showing the verification effect of the Q-MVAE model in the decoupling representation and generation experiment of private information;

[0047] Figure 7 2 is a schematic diagram of the structure of a multimodal variational autoencoder model training system 200 provided in an embodiment of the present invention;

[0048] Figure 8 It is a structural diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0050] Please refer to Figure 1 , Figure 1 The following is a flowchart of a multimodal variational autoencoder model training method according to an embodiment of the present invention, wherein the method comprises the following steps:

[0051] S1. Obtain image data containing multiple modalities, and pair every two images of different modalities to obtain an image pair dataset.

[0052] Specifically, in an embodiment of the present invention, one of the pairs of images in the image pair dataset is defined as a first modality image x and a second modality image y, and a pair of two-modality data obtained by pairing the first modality image x and the second modality image y is (x, y). The two-modality data (x, y) has a shared latent vector z and private latent vectors h corresponding to the first modality image x and the second modality image y, respectively. x 、h y .

[0053] Exemplarily, the image data includes MNIST handwritten digital images and SVHN street number digital images. The images contained in these two digital images have two different modalities. Each digital image of the above two modalities is combined with 20 images of the same digital category of the other modality to form an image pair with the same digital category to obtain the MNIST-SVHN dataset. In an embodiment of the present invention, the MNIST-SVHN dataset is used as the image pair dataset to train a multimodal variational autoencoder model.

[0054] S2. Construct a multimodal variational autoencoder framework model, use the image pair dataset as input of the multimodal variational autoencoder framework model, and train the multimodal variational autoencoder framework model.

[0055] Before describing the model structure of the embodiment of the present invention, the concept of the multimodal variational autoencoder model (VAE) is first described. The goal of VAE is to maximize the log-likelihood function of the joint probability distribution of the data and the marginal probability distribution to obtain the model parameters. For example, in a VAE framework, the observation data x and the corresponding latent variable z are defined, and the generation process is x~p θ (x|z), where z~p(z)=N(0,1), θ is the parameter of the generator network p, then the goal of the multimodal variational autoencoder model is to maximize p(x), p(x) satisfies:

[0056] p(x)=∫p θ (x|z)p(z)dz

[0057] However, p(x) is uncomputable, so the multimodal variational autoencoder model instead calculates the evidence lower bound (ELBO) corresponding to p(x). In general, the evidence lower bound satisfies:

[0058]

[0059] In the expression of the lower bound of evidence, is the negative reconstruction error, D KL is the regularization term, p θ (x|z) is the generator network, q φ (z|x) is the encoder network, where θ and φ are the target model parameters. By substituting the corresponding model parameters into the framework model for training, we can obtain an encoding network and a generation network. The encoding network can represent and reason about the data, and the generation network can randomly generate and reconstruct data.

[0060] In the embodiment of the present invention, please refer to Figure 2 , Figure 2: This is a schematic diagram of the logical structure of the multimodal variational autoencoder framework model provided by an embodiment of the present invention. The embodiment of the present invention specifically constructs a multimodal variational autoencoder model (Q-MVAE) based on quadruple metric loss. The multimodal variational autoencoder framework model is an untrained model framework, which includes a first generator corresponding to the first modal image x and the second modal image y respectively. Second generator The multimodal variational autoencoder framework model also includes the private latent vectors h x and the private latent vector h y The first private information encoder Second private information encoder and first shared information encoders corresponding to the first modality image x and the second modality image y, respectively First shared information encoder

[0061] Furthermore, the multimodal variational autoencoder framework model models the joint probability distribution of the two modal data (x, y) to satisfy the following relationship (1):

[0062]

[0063] In the relation (1), p(z), p(h x )、p(h y ) are the shared latent vector z, the private latent vector h x , the private latent vector h y The prior distribution of , and they all obey the isotropic Gaussian distribution.

[0064] In the embodiment of the present invention, when the multimodal variational autoencoder framework model is trained, for the two-modal data (x, y), its marginal probability distribution p θ (x,y) satisfies the following relationship (2):

[0065]

[0066] For the relationship (2), its integral is not easy to obtain a solution using a normal calculation method. The multimodal variational autoencoder framework model of the embodiment of the present invention uses a variational inference method, using q(z,h x ,h y |x,y) as a variational distribution, the likelihood function p in equation (2) θ (x,y) is transformed:

[0067]

[0068] After that, the variational distribution is decomposed:

[0069]

[0070] q obtained by decomposition of variational distribution z (x, y) is the shared latent vector approximating the posterior probability distribution of the two modal data (x, y).

[0071] When the two modal data (x, y) exist simultaneously, that is, there is no data missing, the multimodal variational autoencoder framework model uses the product of experts (PoE) method to integrate the marginal posteriors of different modalities into a joint posteriors:

[0072]

[0073] According to the above, calculate lnp θ Variational lower bound on (x,y) Variational lower bound The following relationship (3) is satisfied:

[0074]

[0075] On the other hand, when the multimodal variational autoencoder framework model is trained, for the case where the two modal data (x, y) have missing modal data, it is necessary to consider the probability distribution when x or y is missing, and the probability distributions when y is missing and x is missing are:

[0076] p θ (x,z,h x )=p θ (x|z,h x )p(z)p(h x )

[0077] p θ (y,z,h y )=p θ (y|z,h y )p(z)p(h y )

[0078] The corresponding marginal probability distribution p θ (x), p θ (y) satisfies the following relations (4) and (5):

[0079]

[0080]

[0081] Similar to the case where the modal data is complete, in the case where the modal data is missing, the multimodal variational autoencoder framework model still uses the variational inference method, using q(z,h x ,|x)、q(z,h y ,|y) as the variational distribution, the likelihood functions in equations (4) and (5) are transformed to obtain:

[0082]

[0083]

[0084] At the same time, the variational distribution used in the missing case is decomposed:

[0085]

[0086]

[0087] According to the above, calculate lnp θ The variational lower bound of (x) or lnp θ The variational lower bound of (y) Variational lower bound The following relationship (6) is satisfied:

[0088]

[0089] Variational lower bound The following relationship (7) is satisfied:

[0090]

[0091] At this time, the multimodal variational autoencoder framework model can be trained by maximizing equations (3), (6), and (7). However, in order to improve the generation, representation, and decoupling capabilities of modal data, it is necessary to continue to separately infer and represent the shared and private information of each modal data. The shared latent vectors of different modalities need to be aligned in the latent space. Therefore, the embodiment of the present invention introduces the quadruple metric loss to perform latent vector alignment processing.

[0092] S3. Use the quadruple metric loss as the loss function for training the multimodal variational autoencoder framework model until the loss function converges, and output the trained multimodal variational autoencoder model.

[0093] Define the negative samples corresponding to the first modality image x and the second modality image y as x_ and y_ respectively, please refer to Figure 3 , Figure 3is an image sample example provided by an embodiment of the present invention, where x and y are MNIST and SVHN images of digital category 2, respectively, x_ is an MNIST image of digital category 5, and y_ is an SVHN image of digital category 8. It should be noted that x_ and y_ can also be images of any digital category other than 2, and the digital category relationship between x_ and y_ has no correlation with the first modal image x and the second modal image y, and the digital categories of the two images can be the same or different.

[0094] The four-tuple in the four-tuple metric loss is defined as (x, y, x_, y_), and the constraints on the four-tuple metric loss for the loss function convergence are The following relationship (8) is satisfied:

[0095]

[0096] In equation (8), μ x (x) and μ y (y) respectively represent the first shared information encoder The second shared information encoder The first and second max{} terms respectively constrain the metric between x and y to be less than the metric between x and y and the negative samples x_ and y_. The third max{} term constrains the metric between x and y to be less than the metric between any two negative samples of different modal data. α1 and α2 are hyperparameters.

[0097] Combining equations (3), (6), (7), and (8), the final objective function of the multimodal variational autoencoder framework model is The following relationship (9) is satisfied:

[0098]

[0099] Among them, β is a hyperparameter.

[0100] In the case of relational formula (9), the multimodal variational autoencoder framework model constructed by the embodiment of the present invention can be trained to simultaneously consider the two situations of complete and partially missing modal data. In the case of complete modal data, the multimodal variational autoencoder framework model performs product function mixing on the shared latent variable z of different modal data through the expert product function. Since the encoding networks are all Gaussian distributions, they are still Gaussian distributions after mixing. The calculation of their mean and variance has an analytical solution and can be directly used for the training of the multimodal variational autoencoder framework model. In the case of missing modal data, the multimodal variational autoencoder framework model uses the first shared information encoder The second shared information encoder The output result z is used as the shared latent variable z.

[0101] Furthermore, the multimodal variational autoencoder model is optimized using a gradient descent method during training, and Adam is used as an optimizer. Exemplarily, the training parameters used in the embodiment of the present invention when training the multimodal variational autoencoder framework model and outputting the multimodal variational autoencoder model are as follows: experimental training iteration number epoch = 10, batch size batchsize = 128, latent space dimension zdims = 20, the values of the various hyperparameters α1, α2, and β in training are 2, 0.8, and 1500, respectively, and the learning rate lr = 0.0001 of Adam is used as the optimizer. A fully connected neural network is used for MNIST modal image data, and a convolutional neural network is used for SVHN modal image data, wherein the convolution kernel size is 3×3, the convolution step size stride is 2, and the padding value is 1.

[0102] When the final objective function When convergence occurs, the training is completed and the multimodal variational autoencoder model that has completed the training is output.

[0103] For example, the effect diagram of qualitative experimental verification of the trained Q-MVAE model provided by the embodiment of the present invention and the existing MVAE, MMVAE, and DMVAE models in the data cross generation experiment is as follows: Figure 4 The accuracy data of the quantitative experimental verification comparison are shown in Table 1 below.

[0104] Table 1 Accuracy data of data cross generation comparison

[0105]

[0106] Figure 4 In each generated effect diagram, the first row is the known conditional modal sample, and the remaining rows represent the generated missing modal samples with random private information. The shared latent vector of each row is extracted from the corresponding conditional modal sample and the private latent vector is the same. Figure 4 (a)-(d) represent the generation results of MNIST modal data generated from conditional SVHN modal data. By comparing the experimental results of the four models, it can be seen that the Q-MVAE model of the embodiment of the present invention can generate clear images with a random style and the same digital category as the known samples; Table 1 shows the data cross-generation accuracy of Q-MVAE, where each experiment is averaged by 5 independent experiments under different random seeds, and the value in brackets is the overall standard deviation. It can be seen from Table 1 that the cross-generation accuracy of the Q-MVAE model in different directions of the two modalities exceeds that of each comparison model, which is consistent with Figure 4This is consistent with the qualitative generation effect shown in . According to the known conditional samples, Q-MVAE can extract the shared information well and accurately generate other modal data with the shared information.

[0107] The effect diagram of the qualitative experimental verification of the trained Q-MVAE model provided by the embodiment of the present invention and the existing DMVAE model in the data conversion generation experiment is as follows: Figure 5 As shown. Figure 5 It can be seen that the Q-MVAE model can accurately extract the public and private information of the image, and can perform corresponding conversion and generation, and the generation quality is clearer.

[0108] The accuracy data of the trained Q-MVAE model provided by an embodiment of the present invention compared with the existing MVAE, MMVAE, and DMVAE models in multimodal data classification are shown in Table 2 below.

[0109] Table 2 Accuracy data of multimodal data classification comparison

[0110]

[0111] Table 2 shows the experimental results of multimodal classification accuracy on the MNIST-SVHN dataset. Columns 2 and 3 represent classification tests using only unimodal data, while column 4 represents classification tests using both modal data. It can be seen that the Q-MVAE model achieves higher classification accuracy than all the comparison models. This demonstrates that the Q-MVAE model can effectively extract shared information representing data from different modalities. The high accuracy achieved by using a single linear classifier for simultaneous classification of data from different modalities also demonstrates that the proposed quadruple metric loss effectively aligns the shared latent vectors of different modalities in the latent space.

[0112] The verification effect of the trained Q-MVAE model provided by the embodiment of the present invention on the decoupling representation and generation experiment of private information is as follows: Figure 6 As shown, the embodiment of the present invention designs a cross-generation experiment based on the MNIST-SVHN dataset. Under the premise that the values of all dimensions of the private latent vector except a certain dimension remain unchanged, the value of the dimension is linearly changed slightly to observe the image generation effect. Figure 6For cross generation in two directions, the 10 dimensions in the private latent vector are linearly fine-tuned dimension by dimension for data generation. In 5(a), the first row shows that the size of the generated number increases with the slight increase of the value of the first dimension of the private latent vector. It can be considered that the first dimension of the private latent vector has decoupled and learned the generation factor of the font size. The angle of the number in the fifth row changes from the backward angle of the number to the forward angle of the number as the fifth dimension of the private latent vector changes. It can be considered that the fifth dimension has learned the generation factor of the control angle of the number; the tenth row can be considered to have learned the generation factor of the background brightness; from Figure 5 As can be seen in (b), the first row corresponds to the latent vector that learns the factor for the angle of the digit, the third row corresponds to the latent vector that learns the factor for the thickness of the digit font, and the following rows learn the factors for the thickness of the font and different font styles. Figure 6 The effect diagram shows that the Q-MVAE model provided by the embodiment of the present invention has the potential to generate more detailed factor reasoning for image style information.

[0113] The beneficial effect achieved by the present invention lies in proposing a multimodal variational autoencoder model (Q-MVAE) based on quadruple metric loss and its training method. During the training process, shared and private information are modeled separately, and the joint probability distribution of different modalities and the unimodal marginal probability distribution are modeled. At the same time, a loss function is used to constrain the alignment of shared latent vectors in the latent space, thereby improving the data generation and representation capabilities of the model.

[0114] The embodiment of the present invention also provides a multimodal variational autoencoder model training system, please refer to Figure 7 , Figure 7 is a structural diagram of a multimodal variational autoencoder model training system 200 provided in an embodiment of the present invention. The multimodal variational autoencoder model training system 200 includes:

[0115] The data acquisition module 201 is used to acquire image data containing multiple modalities and pair every two images of different modalities to obtain an image pair dataset;

[0116] A model construction module 202 is used to construct a multimodal variational autoencoder framework model, and use the image pair dataset as input of the multimodal variational autoencoder framework model to train the multimodal variational autoencoder framework model;

[0117] The optimization module 203 is configured to use the quadruple metric loss as the loss function for training the multimodal variational autoencoder framework model until the loss function converges, and output the trained multimodal variational autoencoder model.

[0118] The multimodal variational autoencoder model training system 200 can implement the steps in the multimodal variational autoencoder model training method in the above embodiment, and can achieve the same technical effects. Please refer to the description in the above embodiment and will not repeat it here.

[0119] The embodiment of the present invention also provides a computer device, please refer to Figure 8 , Figure 8 3 is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. The computer device 300 includes: a memory 302, a processor 301, and a computer program stored in the memory 302 and executable on the processor 301.

[0120] The processor 301 calls the computer program stored in the memory 302 to execute the steps in the multimodal variational autoencoder model training method provided by the embodiment of the present invention. Figure 1 , specifically including:

[0121] S1. Acquire image data containing multiple modalities and pair every two images of different modalities to obtain an image pair dataset.

[0122] Furthermore, one of the pairs of images in the image pair dataset is defined as a first modality image x and a second modality image y, and a pair of two-modality data obtained by pairing the first modality image x and the second modality image y is (x, y), and the two-modality data (x, y) has a shared latent vector z and private latent vectors h corresponding to the first modality image x and the second modality image y respectively. x 、h y ;

[0123] The multimodal variational autoencoder framework model includes a first generator corresponding to the first modality image x and the second modality image y respectively. Second generator The multimodal variational autoencoder framework model also includes the private latent vectors h x and the private latent vector h y The first private information encoder Second private information encoder and first shared information encoders corresponding to the first modality image x and the second modality image y, respectively First shared information encoder

[0124] S2. Construct a multimodal variational autoencoder framework model, use the image pair dataset as input of the multimodal variational autoencoder framework model, and train the multimodal variational autoencoder framework model.

[0125] Furthermore, the multimodal variational autoencoder framework model models the joint probability distribution of the two modal data (x, y) to satisfy the following relationship (1):

[0126]

[0127] In the relation (1), p(z), p(h x )、p(h y ) are the shared latent vector z, the private latent vector h x , the private latent vector h y The prior distribution of , and they all obey the isotropic Gaussian distribution.

[0128] Furthermore, when the multimodal variational autoencoder framework model is trained, for the two-modal data (x, y), its marginal probability distribution p θ (x,y) satisfies the following relationship (2):

[0129]

[0130] The multimodal variational autoencoder framework model uses variational inference method, using q(z,h x ,h y |x,y) as a variational distribution and calculate lnp θ Variational lower bound on (x,y) The variational lower bound The following relationship (3) is satisfied:

[0131]

[0132] Furthermore, when the multimodal variational autoencoder framework model is trained, for the case where the two modal data (x, y) have missing modal data, its marginal probability distribution p θ (x), p θ (y) satisfies the following relations (4) and (5):

[0133]

[0134]

[0135] The multimodal variational autoencoder framework model uses variational inference method, using q(z,h x ,|x)、q(z,h y ,|y) as the variational distribution and calculate lnp θ The variational lower bound of (x) Or the variational lower bound of lnpθ(y) Variational lower bound The following relationship (6) is satisfied:

[0136]

[0137] Variational lower bound The following relationship (7) is satisfied:

[0138]

[0139] Furthermore, the negative samples corresponding to the first modality image x and the second modality image y are defined as x_ and y_, respectively, and the constraints of the four-tuple metric loss are The following relationship (8) is satisfied:

[0140]

[0141] In equation (8), μ x (x) and μ y (y) respectively represent the first shared information encoder The second shared information encoder The mean of the output results, α1 and α2 are hyper parameters;

[0142] The final objective function of the multimodal variational autoencoder framework model is The following relationship (9) is satisfied:

[0143]

[0144] Among them, β is a hyperparameter.

[0145] S3. Use the quadruple metric loss as the loss function for training the multimodal variational autoencoder framework model until the loss function converges, and output the trained multimodal variational autoencoder model.

[0146] Furthermore, the multimodal variational autoencoder model is optimized using a gradient descent method during training, using Adam as the optimizer.

[0147] The computer device 300 provided in an embodiment of the present invention can implement the steps in the multimodal variational autoencoder model training method in the above embodiment and can achieve the same technical effects. Please refer to the description in the above embodiment and will not repeat it here.

[0148] An embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes and steps in the multimodal variational autoencoder model training method provided in the embodiment of the present invention are implemented, and the same technical effects can be achieved. To avoid repetition, they will not be described here.

[0149] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0150] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0151] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0152] The embodiments of the present invention are described above in conjunction with the accompanying drawings. What is disclosed is only a preferred embodiment of the present invention. However, the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms and equivalent changes without departing from the scope of protection of the purpose of the present invention and the claims, which are all within the protection of the present invention.

Claims

1. A multimodal variational autoencoder model training method, characterized in that: The method comprises the following steps: Acquire image data containing multiple modalities and pair every two images of different modalities to obtain an image pair dataset; Constructing a multimodal variational autoencoder framework model, and using the image pair dataset as input of the multimodal variational autoencoder framework model to train the multimodal variational autoencoder framework model; The quadruple metric loss is used as the loss function for training the multimodal variational autoencoder framework model until the loss function converges, and the trained multimodal variational autoencoder model is output.

2. The multimodal variational autoencoder model training method according to claim 1, wherein: Define one pair of images in the image pair dataset as a first modality image x and a second modality image y, wherein the first modality image x and the second modality image y are paired to obtain a pair of two-modality data (x, y), and the two-modality data (x, y) have a shared latent vector z and private latent vectors h corresponding to the first modality image x and the second modality image y respectively. x 、h y ; The multimodal variational autoencoder framework model includes a first generator corresponding to the first modality image x and the second modality image y respectively. Second generator The multimodal variational autoencoder framework model also includes the private latent vectors h x and the private latent vector h y The first private information encoder Second private information encoder and first shared information encoders corresponding to the first modality image x and the second modality image y, respectively Second shared information encoder 3. The multimodal variational autoencoder model training method according to claim 2, wherein: The multimodal variational autoencoder framework model models the joint probability distribution of the two modal data (x, y) to satisfy the following relationship (1): In the relation (1), p(z), p(h x )、p(h y ) are the shared latent vector z, the private latent vector h x , the private latent vector h y The prior distribution of , and they all obey the isotropic Gaussian distribution.

4. The multimodal variational autoencoder model training method according to claim 3, wherein: When the multimodal variational autoencoder framework model is trained, for the two-modal data (x, y), its marginal probability distribution p θ (x,y) satisfies the following relationship (2): The multimodal variational autoencoder framework model uses variational inference method, using q(z,h x ,h y |x,y) as a variational distribution and calculate lnp θ Variational lower bound on (x,y) The variational lower bound The following relationship (3) is satisfied:

5. The multimodal variational autoencoder model training method according to claim 4, wherein: When the multimodal variational autoencoder framework model is trained, for the case where the two modal data (x, y) have missing modal data, its marginal probability distribution p θ (x), p θ (y) satisfies the following relations (4) and (5): The multimodal variational autoencoder framework model uses variational inference method, using q(z,h x ,|x)、q(z,h y ,|y) as the variational distribution and calculate lnp θ The variational lower bound of (x) Or the variational lower bound of lnpθ(y) Variational lower bound The following relationship (6) is satisfied: Variational lower bound The following relationship (7) is satisfied:

6. The multimodal variational autoencoder model training method according to claim 5, wherein: Define the negative samples corresponding to the first modality image x and the second modality image y as x_, y_ respectively, and the constraints of the four-tuple metric loss The following relationship (8) is satisfied: In equation (8), μ x (x) and μ y (y) respectively represent the first shared information encoder The second shared information encoder The mean of the output results, α1 and α2 are hyper parameters; The final objective function of the multimodal variational autoencoder framework model is The following relationship (9) is satisfied: Among them, β is a hyperparameter.

7. The multimodal variational autoencoder model training method according to claim 1, wherein: The multimodal variational autoencoder model is optimized using the gradient descent method during training, using Adam as the optimizer.

8. A multimodal variational autoencoder model training system, characterized in that: include: The data acquisition module is used to acquire image data containing multiple modalities and pair every two images of different modalities to obtain an image pair dataset; A model construction module is used to construct a multimodal variational autoencoder framework model, and use the image pair dataset as input of the multimodal variational autoencoder framework model to train the multimodal variational autoencoder framework model; An optimization module is used to use the quadruple metric loss as the loss function for training the multimodal variational autoencoder framework model until the loss function converges, and output the trained multimodal variational autoencoder model.

9. A computer device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps in the multimodal variational autoencoder model training method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps in the multimodal variational autoencoder model training method according to any one of claims 1 to 7.

Images