Retrieval method and device based on cross-modal retrieval model

Abstract

The application provides a retrieval method and device based on a cross-modal retrieval model. The method comprises obtaining first cross-modal data and corresponding second cross-modal data to obtain a paired training data set; a cross-modal retrieval model is established, and the cross-modal retrieval model is trained by using the training data set, wherein the cross-modal retrieval model comprises a multi-modal encoder, a discriminator and a memory vector-based modal shared feature extractor, the memory vector-based modal shared feature extractor is used to enhance modal interaction, and a second-order similarity loss is used to enhance modal alignment in the training process; the trained cross-modal retrieval model is used to retrieve the first cross-modal data to be retrieved to generate the second cross-modal data corresponding to the first cross-modal data to be retrieved. Thus, the modal interaction is enhanced in the model structure, the second-order similarity is used to enhance the modal alignment in the training target, and the retrieval effect of the cross-modal retrieval model is effectively improved.

Description

Technical Field

[0001] This invention relates to the field of cross-modal retrieval technology, and particularly to a retrieval method based on a cross-modal retrieval model, a computer-readable storage medium, a computer device, and a retrieval apparatus based on a cross-modal retrieval model. Background Technology

[0002] In related technologies, most existing cross-modal retrieval of text molecules uses two encoders to encode pairs of text and molecules separately, and then uses contrastive learning training objectives for modality alignment. However, due to the large modal differences between text and molecules (one is a sequence and the other is a molecular graph), and the fact that two encoders pre-trained separately on single-modal data are usually used to extract feature representations, contrastive learning alone cannot capture the features shared by modalities well. Furthermore, existing training objectives are based on first-order features of a single instance, ignoring second-order similarity features that contain structural information, which greatly reduces the retrieval performance of cross-modal retrieval models. Summary of the Invention

[0003] This invention aims to at least partially solve one of the technical problems in the aforementioned technologies. To this end, one objective of this invention is to propose a retrieval method based on a cross-modal retrieval model. This method enhances modal interaction in the model structure and strengthens modal alignment using second-order similarity in the training targets, thereby effectively improving the retrieval performance of the cross-modal retrieval model.

[0004] A second objective of this invention is to provide a computer-readable storage medium.

[0005] The third objective of this invention is to provide a computer device.

[0006] The fourth objective of this invention is to propose a retrieval device based on a cross-modal retrieval model.

[0007] To achieve the above objectives, a first aspect of the present invention proposes a retrieval method based on a cross-modal retrieval model. The method includes the following steps: acquiring first cross-modal data and corresponding second cross-modal data to obtain paired training datasets based on the first and second cross-modal data; establishing a cross-modal retrieval model and training the cross-modal retrieval model using the paired training datasets, wherein the cross-modal retrieval model includes a multimodal encoder, a discriminator, and a memory vector-based modality sharing feature extractor, so that the cross-modal retrieval model uses the memory vector-based modality sharing feature extractor to enhance modal interaction and utilizes second-order similarity loss to enhance modal alignment during training; and retrieving the first cross-modal data to be retrieved based on the trained cross-modal retrieval model to generate second cross-modal data corresponding to the first cross-modal data to be retrieved.

[0008] According to the retrieval method based on a cross-modal retrieval model of the present invention, firstly, a first cross-modal data and a corresponding second cross-modal data are acquired to obtain paired training datasets; nextly, a cross-modal retrieval model is established and trained using paired training datasets. The cross-modal retrieval model includes a multimodal encoder, a discriminator, and a memory vector-based modality sharing feature extractor, so that the cross-modal retrieval model uses the memory vector-based modality sharing feature extractor to enhance modal interaction and utilizes second-order similarity loss to enhance modal alignment during training; finally, the trained cross-modal retrieval model is used to retrieve the first cross-modal data to be retrieved, thereby generating the second cross-modal data corresponding to the first cross-modal data to be retrieved. Thus, by strengthening modal interaction in the model structure and enhancing modal alignment using second-order similarity in the training objective, the retrieval performance of the cross-modal retrieval model is effectively improved.

[0009] In addition, the retrieval method based on the cross-modal retrieval model proposed in the above embodiments of the present invention may also have the following additional technical features:

[0010] Optionally, the multimodal encoder includes a text sub-encoder and a molecular sub-encoder; wherein, a pre-trained SciBERT model is used as the initialization parameters of the text sub-encoder, and the input text data is segmented and encoded to obtain feature representations of multiple sub-words; the molecular data is constructed into a molecular graph, with atoms in the molecule as nodes on the graph and chemical bonds in the molecule as edges on the graph, and a graph neural network of GCN is used as the molecular encoder, and pre-trained molecular fingerprints are used as the initial features of atoms to obtain feature representations of multiple atoms.

[0011] Optionally, the discriminator includes a linear layer and is adversarially trained together with the molecular encoder, which serves as a generator.

[0012] Optionally, the memory vector-based modality-sharing feature extractor includes a memory vector library, a cross-attention module, and a linear layer. When the input consists of feature representations of multiple sub-words, the cross-attention module generates multiple modality-sharing feature representations based on the input memory vector and the feature representations of the multiple sub-words. The cross-attention module then performs average pooling on these multiple modality-sharing feature representations before inputting them into the linear layer to obtain a global representation corresponding to the feature representations of the multiple sub-words. When the input consists of feature representations of multiple atoms, the cross-attention module generates multiple modality-sharing feature representations based on the input memory vector and the feature representations of the multiple atoms. The cross-attention module then performs average pooling on these multiple modality-sharing feature representations before inputting them into the linear layer to obtain a global representation corresponding to the feature representations of the multiple atoms.

[0013] Optionally, the total training loss function used during training includes a first-order similarity loss, a second-order similarity loss, a first-order second-order similarity loss, and a second-order second-order similarity loss. The first-order similarity loss is trained based on the contrastive learning loss of triples, where each triple includes an anchor sample, a positive sample from another modality, and a negative sample. The anchor sample and the positive sample come from the same instance, while the negative sample comes from a different instance. The second-order similarity loss is an adversarial training loss implemented using WGAN-GP. Both the first and second-order similarity losses are second-order similarity losses based on KL divergence. The first-order second-order similarity loss brings the text-to-text similarity distribution and the molecule-to-molecule similarity distribution closer together, while the second-order second-order similarity loss uses the text-to-text similarity distribution and the molecule-to-molecule similarity distribution as supervisory signals to guide the modeling of the text-to-molecule similarity distribution and the molecule-to-text similarity distribution.

[0014] To achieve the above objectives, a second aspect of the present invention provides a computer-readable storage medium storing a retrieval program based on a cross-modal retrieval model, which, when executed by a processor, implements the retrieval method based on the cross-modal retrieval model as described above.

[0015] According to an embodiment of the present invention, a computer-readable storage medium stores a retrieval program based on a cross-modal retrieval model. When the retrieval program based on the cross-modal retrieval model is executed by a processor, the retrieval method based on the cross-modal retrieval model described above is implemented. Thus, by strengthening modal interaction in the model structure and enhancing modal alignment using second-order similarity in the training targets, the retrieval effect of the cross-modal retrieval model is effectively improved.

[0016] To achieve the above objectives, a third aspect of the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the retrieval method based on the cross-modal retrieval model described above.

[0017] According to the computer device of the present invention, a retrieval program based on a cross-modal retrieval model is stored in a memory. When the retrieval program based on the cross-modal retrieval model is executed by a processor, the retrieval method based on the cross-modal retrieval model described above is implemented. Thus, by strengthening modal interaction in the model structure and enhancing modal alignment by utilizing second-order similarity in the training targets, the retrieval effect of the cross-modal retrieval model is effectively improved.

[0018] To achieve the above objectives, a fourth aspect of the present invention proposes a retrieval device based on a cross-modal retrieval model, comprising: an acquisition module for acquiring first cross-modal data and corresponding second cross-modal data, so as to obtain paired training datasets based on the first cross-modal data and the second cross-modal data; a training module for establishing a cross-modal retrieval model and training the cross-modal retrieval model using the paired training datasets, wherein the cross-modal retrieval model includes a multimodal encoder, a discriminator, and a memory vector-based modality sharing feature extractor, so that the cross-modal retrieval model uses the memory vector-based modality sharing feature extractor to enhance modal interaction and utilizes second-order similarity loss to enhance modal alignment during training; and a retrieval module for retrieving the first cross-modal data to be retrieved based on the trained cross-modal retrieval model, so as to generate second cross-modal data corresponding to the first cross-modal data to be retrieved; thereby, by strengthening modal interaction in the model structure and enhancing modal alignment in the training objective using second-order similarity, the retrieval performance of the cross-modal retrieval model is effectively improved.

[0019] In addition, the retrieval device based on the cross-modal retrieval model proposed in the above embodiments of the present invention may also have the following additional technical features:

[0020] Optionally, the multimodal encoder includes a text sub-encoder and a molecular sub-encoder; wherein, a pre-trained SciBERT model is used as the initialization parameters of the text sub-encoder, and the input text data is segmented and encoded to obtain feature representations of multiple sub-words; the molecular data is constructed into a molecular graph, with atoms in the molecule as nodes on the graph and chemical bonds in the molecule as edges on the graph, and a graph neural network of GCN is used as the molecular encoder, and pre-trained molecular fingerprints are used as the initial features of atoms to obtain feature representations of multiple atoms.

[0021] Optionally, the memory vector-based modality-sharing feature extractor includes a memory vector library, a cross-attention module, and a linear layer. When the input consists of feature representations of multiple sub-words, the cross-attention module generates multiple modality-sharing feature representations based on the input memory vector and the feature representations of the multiple sub-words. The cross-attention module then performs average pooling on these multiple modality-sharing feature representations before inputting them into the linear layer to obtain a global representation corresponding to the feature representations of the multiple sub-words. When the input consists of feature representations of multiple atoms, the cross-attention module generates multiple modality-sharing feature representations based on the input memory vector and the feature representations of the multiple atoms. The cross-attention module then performs average pooling on these multiple modality-sharing feature representations before inputting them into the linear layer to obtain a global representation corresponding to the feature representations of the multiple atoms. Attached Figure Description

[0022] Figure 1This is a flowchart illustrating a retrieval method based on a cross-modal retrieval model according to an embodiment of the present invention.

[0023] Figure 2 This is a schematic diagram of the overall network structure according to an embodiment of the present invention;

[0024] Figure 3 This is a schematic diagram of the network structure of a modality sharing feature extractor based on memory vectors according to an embodiment of the present invention;

[0025] Figure 4 This is a block diagram of a retrieval device based on a cross-modal retrieval model according to an embodiment of the present invention. Detailed Implementation

[0026] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0027] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the invention to those skilled in the art.

[0028] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0029] Figure 1 This is a flowchart illustrating a retrieval method based on a cross-modal retrieval model according to an embodiment of the present invention, as shown below. Figure 1 As shown, the retrieval method based on a cross-modal retrieval model according to an embodiment of the present invention includes the following steps:

[0030] S101, Obtain the first cross-mode data and the corresponding second cross-mode data, so as to obtain a pair of training datasets based on the first cross-mode data and the second cross-mode data.

[0031] It should be noted that when the first cross-modal data is text, the second cross-modal data is numerator; when the first cross-modal data is numerator, the second cross-modal data is text.

[0032] S102, Establish a cross-modal retrieval model and train the cross-modal retrieval model using paired training datasets. The cross-modal retrieval model includes a multimodal encoder, a discriminator, and a memory vector-based modal sharing feature extractor, so that the cross-modal retrieval model can enhance modal interaction by using the memory vector-based modal sharing feature extractor and enhance modal alignment by using second-order similarity loss during training.

[0033] It should be noted that, as Figure 2 As shown, the cross-modal retrieval model learns the representations of the input text and molecules separately through a multimodal encoder; it performs adversarial training with the molecular encoder, which acts as a generator, through a discriminator to achieve better modal alignment; and it further performs modal interaction based on the outputs of the text sub-encoder and the molecular sub-encoder through a memory vector-based modal sharing feature extractor to extract modal sharing features.

[0034] As one embodiment, the multimodal encoder includes a text sub-encoder and a molecular sub-encoder; wherein, a pre-trained SciBERT model is used as the initialization parameters of the text sub-encoder, and the input text data is segmented and encoded to obtain feature representations of multiple sub-words; the molecular data is constructed into a molecular graph, with atoms in the molecule as nodes on the graph and chemical bonds in the molecule as edges on the graph, and a graph neural network of GCN is used as the molecular encoder, and pre-trained molecular fingerprints are used as the initial features of atoms to obtain feature representations of multiple atoms.

[0035] In other words, a multimodal encoder includes a text sub-encoder and a molecular sub-encoder. The text sub-encoder extracts features from the text sequence, while the molecular sub-encoder extracts features from the molecular graph.

[0036] Specifically, regarding the text sub-encoder, a SciBERT model pre-trained on a large amount of scientific literature is used as the initialization parameters of the text sub-encoder to encode the input text after word segmentation, thereby obtaining feature representations of L sub-words: Regarding the molecular encoder, the molecule is first constructed as a molecular graph, with atoms in the molecule as nodes and chemical bonds in the molecule as edges. A graph neural network based on GCN is used as the molecular encoder, and pre-trained molecular fingerprints are used as the initial features of the atoms. After passing through the molecular encoder, feature representations of K atoms are obtained.

[0037] As one example, the discriminator includes a linear layer and is adversarially trained together with a molecular sub-encoder, which serves as a generator.

[0038] It should be noted that the discriminator is a simple linear layer. Since WGAN-GP is used as the specific implementation method for adversarial training, this discriminator is used to evaluate the Wasserstein distance between the text features and molecular features generated by the two sub-encoders and to maximize the Wasserstein distance during training. The molecular sub-encoder forms an adversarial relationship with the discriminator.

[0039] As one embodiment, the memory vector-based modality-sharing feature extractor includes a memory vector library, a cross-attention module, and a linear layer. When the input consists of feature representations of multiple sub-words, the cross-attention module generates multiple modality-sharing feature representations based on the input memory vector and the feature representations of the multiple sub-words. These multiple modality-sharing feature representations are then averaged and input into the linear layer to obtain a global representation corresponding to the feature representations of the multiple sub-words. When the input consists of feature representations of multiple atoms, the cross-attention module generates multiple modality-sharing feature representations based on the input memory vector and the feature representations of the multiple atoms. These multiple modality-sharing feature representations are then averaged and input into the linear layer to obtain a global representation corresponding to the feature representations of the multiple atoms.

[0040] In other words, the modality-sharing feature extractor based on memory vectors uses a series of trainable memory vectors as query vectors in the cross-attention module to interact with the sub-word features of the text and the atomic features of the molecule, respectively, to extract modality-sharing semantic information. Thus, even if the number of sub-words in the text sequence and the number of atoms in the molecule graph differ greatly, the same number of fine-grained representations will be obtained after mapping with a fixed number of memory vectors. The global representation obtained by average pooling on this basis is more conducive to subsequent training objectives such as contrastive learning.

[0041] Specifically, such as Figure 3 As shown, taking the text modality as an example, the word representation H of the input text... t The modality sharing feature extractor based on memory vectors will be mapped to n modality sharing vectors O. t =Attn(Q, H) t W K H t W v ), where Q∈R n×d W represents the n learnable memory vectors in a memory vector-based modality-sharing feature extractor; K and W V These represent the learnable parameters that map the input features to the key space and value space, respectively; O t ∈R n×d This represents the shared feature representation of n modalities output by the cross-attention module. After average pooling, this representation is fed into a linear layer to obtain the final global representation of the input text: x t=FC(meanPool(O) t Similarly, taking molecular modes as an example, when H is represented by atoms... m When used as input, the final global representation x of the molecule can also be obtained. m .

[0042] As an example, the total training loss function used during training includes a first-order similarity loss, a second-order similarity loss, a first-order second-order similarity loss, and a second-order second-order similarity loss. The first-order similarity loss is trained based on the contrastive learning loss of triples, where each triple includes an anchor sample, a positive sample from another modality, and a negative sample. The anchor sample and the positive sample come from the same instance, while the negative sample comes from a different instance. The second-order similarity loss is an adversarial training loss implemented using WGAN-GP. Both the first and second-order similarity losses are second-order similarity losses based on KL divergence. The first-order second-order similarity loss brings the text-to-text similarity distribution and the molecule-to-molecule similarity distribution closer together, while the second-order second-order similarity loss uses the text-to-text similarity distribution and the molecule-to-molecule similarity distribution as supervisory signals to guide the modeling of the text-to-molecule similarity distribution and the molecule-to-text similarity distribution.

[0043] It should be noted that, in order to effectively train the model, a new second-order similarity loss is added to the first-order similarity loss function, namely the contrastive learning loss and the adversarial training loss; the total training loss function is... in, For the first-order similarity loss, For the second-order similarity loss, For the first and second order similarity loss, This represents the second-order similarity loss.

[0044] Among them, for first-order similarity loss and Cross-modal alignment is achieved by considering only the modal representations of individual instances. Specifically, contrastive learning aims to reduce the distance between positive example pairs while increasing the distance between negative example pairs. Furthermore, adversarial training enables the encoder to generate similar representations for instances from different modalities. Similar to previous work, a triplet-based contrastive learning loss is used for model training. Specifically, each triplet includes an anchor sample and a positive and a negative sample from another modality; the anchor sample and the positive sample come from the same instance, while the negative sample comes from a different instance. This loss can be expressed as: in, and These represent the global feature representations of the text modality and the molecular modality as anchor samples, respectively. and Representative and anchor samples Molecular mode representations belonging to the same and different instances, and Representative and anchor samples Text modal representations belonging to the same and different instances; α represents the marginal distance hyperparameter in this loss function, and d() is the distance function measuring between two samples, which is the Euclidean distance function used in this invention. Additionally, for the loss in adversarial training, the WGAN-GP implementation is used: Where D represents the discriminator, p t and p m These represent the characteristic distributions of text and molecules, respectively.

[0045] Among them, for second-order similarity loss and Since the aforementioned contrastive learning and adversarial training are based on features of a single instance, the second-order similarity loss considers the similarity between different instances. It can serve as a structural feature reflecting the neighbor relationships of samples in the feature space, aiming to ensure that the similarity between pairs of instances observed from different modalities is close, thereby enhancing the alignment between different modalities. Specifically, given a training batch B, which includes multiple pairs of text and molecule data, four similarity matrices are first calculated based on the global representations of the text and molecules. For example, for the i-th and j-th instances in B, their text-to-text similarity, molecule-to-molecule similarity, text-to-molecule similarity, and molecule-to-text similarity are calculated. Furthermore, for the i-th instance, four similarity distributions between it and each instance in the current training batch B can be calculated: text-to-text similarity score (TTSD). Molecular-to-molecular similarity distribution (MMSD) Text-to-Molecular Similarity Distribution (MTSD) Molecular-to-text similarity distribution (TMSD)(TTSD) Taking TTSD as an example, the matrix is ​​calculated as follows:

[0046]

[0047] Then, second-order similarity loss based on KL divergence. and Modality alignment is enhanced by improving the consistency of different similarity matrices. In this study, the two unimodal similarity matrices, TTSD and MMSD, are brought closer together. The unimodal similarity matrix reflects the structural relationships between different instances in the current modality, and bringing these second-order similarity features closer together is beneficial for aligning the two modalities. The unimodal similarity matrices TTSD and MMSD are used as supervision signals to guide the modeling of cross-modal similarity matrices MTSD and TMSD. This is because the unimodal similarity matrices have already been pre-trained on a large amount of unimodal data and are more accurate than the cross-modal similarity matrices. Furthermore, using continuous numerical similarity as supervision signals provides richer information than simply determining whether a match is correct in contrastive learning. The formulas for these two second-order similarity losses are as follows:

[0048]

[0049]

[0050] Therefore, by designing a modality sharing feature extractor based on memory vectors in the model structure, the fine-grained feature vectors of the encoder output for text and molecules are further interacted, which makes up for the significant modality gap between the two types of data. Furthermore, a second-order similarity loss is proposed from different perspectives, which not only enriches the supervision signal but also utilizes the structural information between instances, thereby improving the degree of modality alignment.

[0051] S103, based on the trained cross-modal retrieval model, retrieve the first cross-modal data to be retrieved, so as to generate the second cross-modal data corresponding to the first cross-modal data to be retrieved.

[0052] It should be noted that after the acquired first cross-modal data to be retrieved is input into the trained cross-modal retrieval model, the trained cross-modal retrieval model is used to retrieve the second cross-modal data dataset so that the pairwise similarity can be calculated by matrix multiplication and the second cross-modal data with high similarity can be output, thereby obtaining the second cross-modal data corresponding to the first cross-modal data.

[0053] In summary, the retrieval method based on a cross-modal retrieval model according to embodiments of the present invention firstly acquires first cross-modal data and corresponding second cross-modal data to obtain paired training datasets based on the first and second cross-modal data; then, a cross-modal retrieval model is established and trained using paired training datasets. The cross-modal retrieval model includes a multimodal encoder, a discriminator, and a memory vector-based modal sharing feature extractor, enabling the cross-modal retrieval model to enhance modal interaction using the memory vector-based modal sharing feature extractor and enhance modal alignment using second-order similarity loss during training; finally, the trained cross-modal retrieval model is used to retrieve the first cross-modal data to be retrieved, generating the second cross-modal data corresponding to the first cross-modal data to be retrieved. Thus, by strengthening modal interaction in the model structure and enhancing modal alignment using second-order similarity in the training objective, the retrieval performance of the cross-modal retrieval model is effectively improved.

[0054] In addition, this embodiment of the invention also proposes a computer-readable storage medium storing a retrieval program based on a cross-modal retrieval model, which, when executed by a processor, implements the retrieval method based on the cross-modal retrieval model as described above.

[0055] According to an embodiment of the present invention, a computer-readable storage medium stores a retrieval program based on a cross-modal retrieval model. When the retrieval program based on the cross-modal retrieval model is executed by a processor, the retrieval method based on the cross-modal retrieval model described above is implemented. Thus, by strengthening modal interaction in the model structure and enhancing modal alignment using second-order similarity in the training targets, the retrieval effect of the cross-modal retrieval model is effectively improved.

[0056] In addition, this invention also proposes a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the retrieval method based on the cross-modal retrieval model described above.

[0057] According to the computer device of the present invention, a retrieval program based on a cross-modal retrieval model is stored in a memory. When the retrieval program based on the cross-modal retrieval model is executed by a processor, the retrieval method based on the cross-modal retrieval model described above is implemented. Thus, by strengthening modal interaction in the model structure and enhancing modal alignment by utilizing second-order similarity in the training targets, the retrieval effect of the cross-modal retrieval model is effectively improved.

[0058] To implement the above embodiments, this invention also proposes a retrieval device based on a cross-modal retrieval model, such as... Figure 4 As shown, the retrieval device based on the cross-modal retrieval model includes: an acquisition module 10, a training module 20, and a retrieval module 30.

[0059] The acquisition module 10 is used to acquire the first cross-modal data and the corresponding second cross-modal data, so as to obtain a pair of training datasets based on the first and second cross-modal data. The training module 20 is used to build a cross-modal retrieval model and train the cross-modal retrieval model using the pair of training datasets. The cross-modal retrieval model includes a multimodal encoder, a discriminator, and a memory vector-based modal sharing feature extractor, so that the cross-modal retrieval model can enhance modal interaction by using the memory vector-based modal sharing feature extractor and enhance modal alignment by using second-order similarity loss during training. The retrieval module 30 is used to retrieve the first cross-modal data to be retrieved based on the trained cross-modal retrieval model, so as to generate the second cross-modal data corresponding to the first cross-modal data to be retrieved. Thus, by strengthening modal interaction in the model structure and enhancing modal alignment by using second-order similarity in the training objective, the retrieval effect of the cross-modal retrieval model is effectively improved.

[0060] As one embodiment, the multimodal encoder includes a text sub-encoder and a molecular sub-encoder; wherein, a pre-trained SciBERT model is used as the initialization parameters of the text sub-encoder, and the input text data is segmented and encoded to obtain feature representations of multiple sub-words; the molecular data is constructed into a molecular graph, with atoms in the molecule as nodes on the graph and chemical bonds in the molecule as edges on the graph, and a graph neural network of GCN is used as the molecular encoder, and pre-trained molecular fingerprints are used as the initial features of atoms to obtain feature representations of multiple atoms.

[0061] As one example, the discriminator includes a linear layer and is adversarially trained together with a molecular sub-encoder, which serves as a generator.

[0062] As one embodiment, the memory vector-based modality-sharing feature extractor includes a memory vector library, a cross-attention module, and a linear layer. When the input consists of feature representations of multiple sub-words, the cross-attention module generates multiple modality-sharing feature representations based on the input memory vector and the feature representations of the multiple sub-words. These multiple modality-sharing feature representations are then averaged and input into the linear layer to obtain a global representation corresponding to the feature representations of the multiple sub-words. When the input consists of feature representations of multiple atoms, the cross-attention module generates multiple modality-sharing feature representations based on the input memory vector and the feature representations of the multiple atoms. These multiple modality-sharing feature representations are then averaged and input into the linear layer to obtain a global representation corresponding to the feature representations of the multiple atoms.

[0063] As an example, the total training loss function used during training includes a first-order similarity loss, a second-order similarity loss, a first-order second-order similarity loss, and a second-order second-order similarity loss. The first-order similarity loss is trained based on the contrastive learning loss of triples, where each triple includes an anchor sample, a positive sample from another modality, and a negative sample. The anchor sample and the positive sample come from the same instance, while the negative sample comes from a different instance. The second-order similarity loss is an adversarial training loss implemented using WGAN-GP. Both the first and second-order similarity losses are second-order similarity losses based on KL divergence. The first-order second-order similarity loss brings the text-to-text similarity distribution and the molecule-to-molecule similarity distribution closer together, while the second-order second-order similarity loss uses the text-to-text similarity distribution and the molecule-to-molecule similarity distribution as supervisory signals to guide the modeling of the text-to-molecule similarity distribution and the molecule-to-text similarity distribution.

[0064] It should be noted that the above description and examples of the retrieval method based on the cross-modal retrieval model also apply to the retrieval device based on the cross-modal retrieval model in this embodiment, and will not be repeated here.

[0065] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0066] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0067] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0068] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0069] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0070] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0071] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

[0072] In the description of this invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0073] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0074] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "over," and "on top" of the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0075] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms should not be construed as necessarily referring to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0076] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A retrieval method based on a cross-modal retrieval model, characterized in that, Includes the following steps: Obtain the first cross-modal data and the corresponding second cross-modal data so as to obtain a pair of training datasets based on the first cross-modal data and the second cross-modal data; A cross-modal retrieval model is established and trained using the paired training datasets. The cross-modal retrieval model includes a multimodal encoder, a discriminator, and a memory vector-based modal shared feature extractor, so that the cross-modal retrieval model can enhance modal interaction using the memory vector-based modal shared feature extractor and enhance modal alignment using second-order similarity loss during training. The trained cross-modal retrieval model is used to retrieve the first cross-modal data to be retrieved, so as to generate the second cross-modal data corresponding to the first cross-modal data to be retrieved. The multimodal encoder includes a text sub-encoder and a molecular sub-encoder; The pre-trained SciBERT model is used as the initialization parameter of the text sub-encoder, and the input text data is segmented and encoded to obtain the feature representation of multiple sub-words. Molecular data is constructed into a molecular graph, with atoms in the molecule as nodes and chemical bonds in the molecule as edges. A graph neural network (GCN) is used as the molecular encoder, and pre-trained molecular fingerprints are used as the initial features of atoms to obtain feature representations of multiple atoms. The modality sharing feature extractor based on memory vectors includes a memory vector library, a cross-attention module, and a linear layer. When the input consists of feature representations of multiple sub-words, the cross-attention module generates multiple modality-shared feature representations based on the input memory vector and the feature representations of multiple sub-words. After average pooling, the multiple modality-shared feature representations are input into the linear layer to obtain the global representations corresponding to the feature representations of the multiple sub-words. When the input is a feature representation of multiple atoms, the cross-attention module generates a feature representation shared by multiple modalities based on the input memory vector and the feature representation of multiple atoms, and then performs average pooling on the feature representation shared by multiple modalities before inputting it into the linear layer to obtain the global representation corresponding to the feature representation of multiple atoms.

2. The retrieval method based on a cross-modal retrieval model as described in claim 1, characterized in that, The discriminator includes a linear layer and is adversarially trained together with the molecular encoder, which serves as a generator.

3. The retrieval method based on a cross-modal retrieval model as described in claim 2, characterized in that, The total training loss function used during training includes the first-order similarity loss, the second-order similarity loss, the first-second-order similarity loss, and the second-second-order similarity loss. The first-order similarity loss is trained based on the contrastive learning loss of triples. Triples include anchor samples, positive samples of another modality, and negative samples. The anchor samples and positive samples come from the same instance, while the negative samples come from different instances. The second-order similarity loss is the loss for adversarial training, implemented using WGAN-GP; Both the first and second-order similarity losses are second-order similarity losses based on KL divergence. The first second-order similarity loss brings the text-to-text similarity distribution and the molecule-to-molecule similarity distribution closer together. The second second-order similarity loss uses the text-to-text similarity distribution and the molecule-to-molecule similarity distribution as supervision signals to guide the modeling of the text-to-molecule similarity distribution and the molecule-to-text similarity distribution.

4. A computer-readable storage medium, characterized in that, It stores a retrieval program based on a cross-modal retrieval model, which, when executed by a processor, implements the retrieval method based on a cross-modal retrieval model as described in any one of claims 1-3.

5. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the retrieval method based on the cross-modal retrieval model as described in any one of claims 1-3.

6. A retrieval apparatus based on a cross-modal retrieval model, characterized by, include: The acquisition module is used to acquire the first cross-mode data and the corresponding second cross-mode data, so as to obtain a pair of training datasets based on the first cross-mode data and the second cross-mode data; The training module is used to establish a cross-modal retrieval model and train the cross-modal retrieval model using the paired training datasets. The cross-modal retrieval model includes a multimodal encoder, a discriminator, and a memory vector-based modal shared feature extractor, so that the cross-modal retrieval model can enhance modal interaction by using the memory vector-based modal shared feature extractor and enhance modal alignment by using second-order similarity loss during training. The retrieval module is used to retrieve the first cross-modal data to be retrieved based on the trained cross-modal retrieval model, so as to generate the second cross-modal data corresponding to the first cross-modal data to be retrieved; The multimodal encoder includes a text sub-encoder and a molecular sub-encoder; The pre-trained SciBERT model is used as the initialization parameter of the text sub-encoder, and the input text data is segmented and encoded to obtain the feature representation of multiple sub-words. Molecular data is constructed into a molecular graph, with atoms in the molecule as nodes and chemical bonds in the molecule as edges. A graph neural network (GCN) is used as the molecular encoder, and pre-trained molecular fingerprints are used as the initial features of atoms to obtain feature representations of multiple atoms. The modality sharing feature extractor based on memory vectors includes a memory vector library, a cross-attention module, and a linear layer. When the input consists of feature representations of multiple sub-words, the cross-attention module generates multiple modality-shared feature representations based on the input memory vector and the feature representations of multiple sub-words. After average pooling, the multiple modality-shared feature representations are input into the linear layer to obtain the global representations corresponding to the feature representations of the multiple sub-words. When the input is a feature representation of multiple atoms, the cross-attention module generates a feature representation shared by multiple modalities based on the input memory vector and the feature representation of multiple atoms. The feature representation shared by multiple modalities is then averaged and input into the linear layer to obtain the global representation corresponding to the feature representation of multiple atoms.

Images