Cross-modal entity recall method and device based on modal contrast learning
Through modal comparison learning, the cross-modal entity recall method solves the problems of high spatial time complexity and slow retrieval speed of cross-modal entity retrieval in the existing technology, and achieves rapid recall of entities and reduces space consumption. on the double tower model.
Patent Information
- Application Number
- CN202310643813.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-01
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2043-06-01
Smart Images

Figure CN119066217A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of cross-modal entity retrieval, and in particular to a cross-modal entity recall method and device based on modality contrast learning. Background Art
[0002] Cross-modal retrieval refers to information retrieval constructed between different modalities. The most common cross-modal retrieval is to return relevant text information from a candidate library given an image, or to return relevant images from a candidate library given a text. Entity retrieval refers to returning the entity corresponding to a query from a knowledge base given a query.
[0003] The current cross-modal entity retrieval technology is in an early stage of exploration. Traditional cross-modal retrieval technology can only perform some generalized retrieval, and cannot truly recognize entity granularity. Due to the development of multimodal technology and network big data, some technical methods for cross-modal recognition and retrieval have emerged recently. However, most of the current cross-modal entity retrieval technologies are based on pre-trained multimodal models combined with specific training tasks for entity retrieval, such as fine-tuning models such as entity mask reconstruction.
[0004] Fine-tuning the pre-trained model based on the entity masking task. On the one hand, the entity masking task requires the construction of an entity-scale vocabulary for training. However, the scale of entities in the knowledge base is large, which will consume a lot of space. At the same time, there are also very high requirements for the scale of training data, and it is necessary to ensure that different entities appear many times in the data set. On the other hand, this pre-training task is suitable for interaction-based models and cannot be directly applied to the currently commonly used dual-tower structure. Therefore, fine-tuning the pre-trained model based on the entity masking task has the problems of high spatial and temporal complexity, dependence on data scale, and cannot be directly applied to the commonly used dual-tower retrieval framework. In addition, the entity mask reconstruction task is generally based on an interaction-based model, which will cause slow retrieval speed in the entity retrieval stage. Summary of the invention
[0005] Therefore, the purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a cross-modal entity recall method and device based on modality contrast learning.
[0006] The objective of the present invention is achieved through the following technical solutions:
[0007] According to a first aspect of the present invention, a cross-modal entity recall method based on modal contrast learning is provided, comprising: T1, obtaining an entity to be queried, and extracting a multidimensional entity vector of the entity to be queried using a trained multimodal encoding neural network and a preset normalization method, wherein the multimodal encoding neural network comprises a text encoding model for extracting vectors for input entities in text modality and an image encoding model for extracting vectors for input entities in image modality, and is obtained according to same-modal contrast learning and cross-modal contrast learning; T2, calculating the similarity between the multidimensional entity vector to be queried and the multidimensional in-library entity vectors of each in-library entity stored in an index library, wherein the multidimensional in-library entity vectors of each in-library entity are extracted in advance based on the trained multimodal encoding neural network and the preset normalization method; T3, based on the similarity, recalling multiple in-library entities from the index library as candidate entities.
[0008] In some embodiments of the present invention, the same-modality contrastive learning is used to increase the distance between vectors output by encoding models of the corresponding modality for different entities, and the cross-modality contrastive learning is used to decrease the distance between vectors output by encoding models of different modalities for related entities.
[0009] In some embodiments of the present invention, the trained multimodal encoding neural network is trained in the following manner: multiple first samples of text modality and multiple second samples of image modality are obtained, each first sample is provided with a second sample of a different modality but corresponding to an entity and an association relationship is established between the two, different first samples correspond to different entities, and different second samples correspond to different entities; the first sample is input into a text encoding model to obtain a vector of the first sample; the second sample is input into an image encoding model to obtain a vector of the second sample; the distance between the vectors of different first samples and the distance between the vectors of different second samples are used to perform same-modality comparative learning, and based on the association relationship, the distance between the vector of the first sample and the vector of the second sample corresponding to the related entity is used to perform cross-modal comparative learning.
[0010] In some embodiments of the present invention, during training, the loss value is configured to be negatively correlated with the distance between vectors between different first samples and the distance between vectors between different second samples, and the loss value is configured to be positively correlated with the distance between the vector of the first sample and the vector of the second sample corresponding to the related entity.
[0011] In some embodiments of the present invention, homomodal contrastive learning constructs a first matrix for multiple first samples and a second matrix for multiple second samples, wherein the diagonal elements of the first matrix and the second matrix represent similarities determined based on vectors of homomodal positive samples, and the off-diagonal elements represent similarities determined based on vectors of homomodal negative samples; cross-modal contrastive learning constructs a third matrix for the first samples and the second samples corresponding to related entities, wherein the diagonal elements of the third matrix represent similarities of cross-modal positive samples, and the off-diagonal elements represent similarities determined based on vectors of cross-modal negative samples; and the loss value calculated in the following manner updates the parameters of the multimodal encoding neural network:
[0012]
[0013] in, represents the similarity of a single cross-modal positive sample, τ represents the preset hyperparameter, exp(·) represents the mean, Represents the similarity of a single homomodal negative sample or cross-modal negative sample, k∈n, j∈n.
[0014] In some embodiments of the present invention, in step T1, if the entity to be queried is text, the text encoding model is used to encode the entity to be queried to obtain a multidimensional vector and the multidimensional vector is L2 normalized to obtain a multidimensional entity vector of the entity to be queried; if the entity to be queried is an image, the image encoding model is used to encode the entity to be queried to obtain a multidimensional vector and the multidimensional vector is L2 normalized to obtain a multidimensional entity vector of the entity to be queried.
[0015] In some embodiments of the present invention, in step T2, the index library includes a text entity index library constructed based on a text encoding model and an image entity index library constructed based on an image encoding model, and the calculation of the similarity between the multidimensional entity vector to be queried and the multidimensional entity vectors of each entity in the library stored in the index library includes: if the entity to be queried is text, calculating the similarity between the multidimensional entity vector to be queried and the multidimensional entity vectors of each entity in the library stored in the image entity index library; if the entity to be queried is an image, calculating the similarity between the multidimensional entity vector to be queried and the multidimensional entity vectors of each entity in the library stored in the text entity index library.
[0016] According to a second aspect of the present invention, there is provided an entity recall device for implementing the method described in the first aspect, the device comprising: a text encoding module, for extracting a multidimensional entity vector of the entity to be queried from an entity in text mode using a text encoding model and a preset normalization method; an image encoding module, for extracting a multidimensional entity vector of the entity to be queried from an entity in image mode using an image encoding model and a preset normalization method; an index library construction module, for constructing a text entity index library storing a plurality of multidimensional text entity vectors using the text encoding module and an image entity index library storing a plurality of multidimensional image entity vectors using the image encoding module; a similarity calculation module, for calculating the similarity between the multidimensional entity vector to be queried and the multidimensional in-library entity vectors of each in-library entity stored in the index library; and an entity recall module, for recalling a plurality of in-library entities from the index library as candidate entities based on the similarity calculated by the similarity calculation module.
[0017] According to a third aspect of the present invention, there is provided an electronic device comprising: one or more processors; and a memory, wherein the memory is used to store executable instructions; the one or more processors are configured to implement the steps of the method described in the first aspect by executing the executable instructions.
[0018] Compared with the prior art, the advantages of the present invention are: a cross-modal entity recall method based on modality contrast learning proposed in the present invention, compared with the traditional entity mask reconstruction task, the present invention not only introduces contrast learning within the modality and contrast learning between modalities, which shortens the distance between related entity pairs in the modalities in space, but also increases the distance between different entities in the modality in space, ensuring the uniform distribution of entities in space, thereby realizing rapid retrieval of entities; the present invention also calculates entity vectors in advance based on the dual-tower model combined with the common vector recall model and stores them offline, so that during online retrieval, only the similarity between the entity to be queried and the entity vector in the index library needs to be calculated to quickly recall the entity, reducing space consumption. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:
[0020] Figure 1 The figure is a flow chart of a method for training a multimodal encoding neural network according to an embodiment of the present invention.
[0021] Figure 2 Schematic diagram of a cross-modal entity retrieval method based on modality contrast learning according to an embodiment of the present invention. DETAILED DESCRIPTION
[0022] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail by specific embodiments below. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0023] As mentioned in the background technology section, most of the existing technologies perform entity retrieval on a pre-trained multimodal model in combination with a specific training task. When performing entity retrieval, it is necessary to build an entity-scale vocabulary for training, and the entity-scale vocabulary needs to ensure that each different entity appears in the data set with a high frequency, so a very large space consumption will be generated in this process; and the pre-trained task cannot be applied to the common dual-tower model and is only applicable to the interaction-based model. When performing entity retrieval under the interaction-based model, the retrieval speed will be slow.
[0024] In order to solve the problems of slow retrieval speed, large space consumption and inability to be directly applied to the dual-tower model in the prior art, the present invention conducts cross-modal entity retrieval research. During the research process, it is found that the problems of slow retrieval speed, large space consumption and inability to be directly applied to the dual-tower model to achieve fast retrieval are caused by the entity mask training task. Therefore, the present invention further studies the entity vector and finds that fast entity retrieval can be achieved by combining the dual-tower model with the modal contrast learning task, and the space consumption can be effectively reduced. Specifically, in the process of entity recall, cross-modal contrast learning that can shorten the distance between related entity pairs between modalities in space and same-modal contrast learning that can increase the distance between different entities in the modality in space are introduced to train the multimodal encoding neural network used to extract entity vectors. This training method greatly reduces the use of space; based on the trained multimodal encoding neural network, the entity vector is stored and an index library is constructed, so that the similarity between the entity to be queried and the entity vector in the index library can be quickly calculated during the entity recall process, so as to achieve fast entity recall and reduce space consumption.
[0025] In order to better understand the present invention, the present invention is introduced from five parts: model structure, training samples, training process, building an index library and application scenarios in combination with the accompanying drawings and examples.
[0026] 1. Model Structure
[0027] First, the model structure adopted by the present invention is a dual-tower model including two encoding neural networks, including: a text encoding neural network for encoding text entities and an image encoding neural network for encoding image entities. Among them, the text encoding neural network includes: a text encoding model and a normalization module; the image encoding neural network includes: an image encoding model and a normalization module.
[0028] According to one embodiment of the present invention, the text encoding model can use the Bert model to encode the text entity to obtain a multidimensional vector of the text entity. It should be noted that the Bert model here is only one of the models for encoding text entities in the embodiment of the present invention, but it is not limited to this. It can also be used to encode text entities: CLIP-Roberta model, ALBert model and XLNET model; these models have some similarities with the Bert model in the performance of encoding text entities, but the CLIP-Roberta model can obtain a better model based on the same training data compared with the Bert model; the ALBert model uses the two methods of Embedding matrix decomposition and parameter sharing at each level to make the model slim without reducing the effect of the model too much; therefore, in the actual application process, the corresponding model can be selected according to the text encoding requirements to process the text, which is not limited or elaborated here.
[0029] According to one embodiment of the present invention, the image coding model can use a ViT (Vision Transformer) model (such as a CLIP-Vit model) to encode the image entity to obtain a multidimensional vector of the image entity. When using the ViT model, those skilled in the art know that the size of the input image needs to be processed to meet the form required for the input of the Transformer model. It should be noted that the ViT model here is only one of the models for encoding image entities in the embodiment of the present invention, but it is not limited to this. The image entity can also be encoded by: RESNET model and MLP-Mixer model; these two models have some similarities with the ViT model in the process of encoding and processing images and processing capabilities, especially the MLP-Mixer model, which needs to process the size of the input image to meet the form required for the input of the model; and the RESNET model is more strictly in accordance with the process of extracting global features from local features than the ViT model; therefore, in the actual application process, the corresponding model can be selected according to the encoding requirements of the image to process the image, and no limitation or excessive elaboration is made here.
[0030] The normalization module is used to normalize the multidimensional vector of the text entity or the multidimensional vector of the image entity to obtain the multidimensional text entity vector and the multidimensional image entity vector. According to one embodiment of the present invention, the multidimensional vector of the text entity and the multidimensional vector of the image entity can be normalized by L2 normalization. For example: the vector x(x1, x2, ..., x n ) is L2 normalized, and the L2 norm of vector x is defined as: To normalize x to the unit L2 norm, that is, to establish a mapping from x to x' so that x' is a vector with L2 norm of 1, then:
[0031]
[0032] Therefore, we can get the L2 normalized vector
[0033] 2. Training Samples
[0034] Secondly, in the process of implementing the cross-modal entity recall method based on modality contrast learning proposed in the present invention, it is necessary to train the text encoding model and the image encoding model in the model structure of the present invention. Before training, it is necessary to organize training samples.
[0035] In order to shorten the distance between related entity pairs between modalities and increase the distance between different entities within a modality in space, and ensure the uniform distribution of entities in space, the present invention introduces cross-modal contrastive learning and same-modal contrastive learning in the part of obtaining training samples. The same-modal contrastive learning is used to increase the distance between vectors output by the encoding model of the corresponding modality for different entities, and the cross-modal contrastive learning is used to shorten the distance between vectors output by the encoding models of different modalities for related entities.
[0036] According to an embodiment of the present invention, an existing data set (such as WIKIPerson, visual person link data set) can be used to construct training samples. However, the organization form of the training samples needs to meet the following requirements: multiple first samples of text modality and multiple second samples of image modality are obtained, each first sample is provided with a second sample of different modality but corresponding to an entity, and an association relationship is established between the two, different first samples correspond to different entities, and different second samples correspond to different entities.
[0037] According to an example of the present invention, when the model is trained in batches, it is assumed that 5 text entity samples and 5 image entity samples are input in each batch as an example to illustrate the organization of the training samples. Each batch of samples includes text entity samples A, B, C, D, E and image entity samples a, b, c, d, e, wherein the upper and lower letters of different modal samples represent samples of a pair of related entities, and different letters in the samples of the same modality are different entities. For example, it is assumed that the entities corresponding to text entity sample A and image entity sample a are related (for example, they are all cats), and the entities corresponding to text entity sample B and image entity sample b are related (for example, they are all dogs), and the rest are similar. In this way, in the same modality, text entity samples A, B, C, D, E can be used to guide same-modality contrast learning, and the association relationship between text entity sample A and image entity sample a guides cross-modality contrast learning.
[0038] 3. Training Process
[0039] Based on the training samples obtained in the second part above, the loss value of the loss function calculation matrix is used to reversely update the parameters of the multimodal encoding neural network so that the text encoding model and the image encoding model can be trained.
[0040] According to one embodiment of the present invention, Figure 1 As shown, a method for training a multimodal encoding neural network (or a modality contrast learning method based on negative examples within a batch) is shown. The steps of the method are as follows:
[0041] Step S1: Obtain a batch of text entity samples, use the Bert model to standardize the batch of text entity samples to obtain a token sequence, and input the token sequence into a text encoding model to obtain a multidimensional vector of n text entity samples;
[0042] Step S2: Input the multidimensional vectors of n text entity samples into the normalization module, and use the normalization function to normalize the multidimensional vectors of n text entity samples into multidimensional text entity vectors; schematically, a text entity sample is passed through the text encoding model to obtain a one-dimensional vector Ft(0.5, 0.6, 0.7), and the one-dimensional vector Ft is L2 normalized to obtain
[0043] Step S3: obtaining a batch of image entity samples, encoding the batch of image entity samples using the Vi T model, and obtaining a multidimensional vector of n image entity samples;
[0044] Step S4: Input the multidimensional vectors of n image entity samples into the normalization module, and use the normalization function to normalize the multidimensional vectors of n image entity samples into multidimensional image entity vectors; schematically, an image entity sample is encoded by the image coding model to obtain a one-dimensional vector F i(0.2, 0.3, 0.7), and the one-dimensional vector F i is L2 normalized to obtain
[0045] Step S5: For cross-modal contrastive learning, with the help of matrix product function, the vector Ft' is multiplied by the transpose of the vector Fi' to obtain a first matrix; for same-modal contrastive learning, the vector Ft' is multiplied by the transpose of the vector Ft' to obtain a second matrix, the vector Fi' is multiplied by the transpose of the vector Fi' to obtain a third matrix, and the first matrix, the second matrix and the third matrix are sequentially concatenated to obtain a fourth matrix.
[0046] Step S6: Based on the fourth matrix obtained by concatenation, the loss value of the fourth matrix is calculated using the loss function, and the parameters of the multimodal encoding neural network are reversely updated according to the calculated loss value to train the text encoding model and the image encoding model.
[0047] According to an example of the present invention, the previous example is still used to continue the description. Assume that the text entity sample A is encoded by the text encoding model to obtain the multidimensional vector of the text entity sample A, and then the multidimensional vector of the text entity sample A is processed by the normalization module to obtain the multidimensional entity vector of the text entity sample A; the image entity sample a is encoded by the image encoding model to obtain the multidimensional vector of the image entity sample a, and then the multidimensional vector of the image entity sample a is processed by the normalization module to obtain the multidimensional entity vector of the image entity sample a; its text entity samples (B, C, D, E) and image entity samples (b, c, d, e) are processed in the above manner, which will not be repeated here. The multidimensional entity vectors of the text entity samples (A, B, C, D, E) and the multidimensional entity vectors of the image entity samples (a, b, c, d, e) are subjected to cross-modal contrast learning and same-modal contrast learning through the contrast learning module, and the cross-modal matrix (first matrix) is obtained respectively:
[0048]
[0049] Same modality text matrix (second matrix):
[0050]
[0051] Same modality image matrix (third matrix)
[0052]
[0053] The elements in the first matrix, the second matrix and the third matrix are the similarities of the vectors of the corresponding samples. The first matrix, the second matrix and the third matrix are concatenated to form the fourth matrix.
[0054] According to one embodiment of the present invention, a homomodal matrix is obtained by homomodal contrast learning based on the distance between vectors of different first samples and the distance between vectors of different second samples, and its diagonal elements are the similarities determined by the vectors of homomodal positive samples, and the off-diagonal elements are the similarities determined by the vectors of homomodal negative samples; a cross-modal matrix is obtained by cross-modal contrast learning based on the association relationship using the distance between the vector of the first sample and the vector of the second sample corresponding to the related entity, and its diagonal elements are the similarities determined by the vectors of cross-modal positive samples, and the off-diagonal elements are the similarities determined by the vectors of cross-modal negative samples. For example, for the homomodal text matrix, homomodal image matrix and cross-modal matrix obtained above, for the homomodal text matrix, its diagonal elements p A,Ais the similarity of <text A, text A>, and the off-diagonal element p A,B is the similarity of <text A, text B>; for the same-modal image matrix, its diagonal element p a,a is the similarity of <image a, image a>, and the off-diagonal element p a,d is the similarity of <image a, image d>; for the cross-modal matrix, the diagonal element p A,a is the similarity of <text A, image a>, and the off-diagonal element p B,c is the similarity of <text B, image c>. The loss value is calculated as follows:
[0055]
[0056] in, represents the similarity of a single cross-modal positive sample, τ represents the preset hyperparameter, exp(·) represents the mean, Represents the similarity of a single homomodal negative sample or cross-modal negative sample, k∈n, j∈n.
[0057] The parameters of the multimodal encoding neural network are updated inversely based on the calculated loss value so that the text encoding model and the image encoding model are trained.
[0058] According to another embodiment of the present invention, a cross entropy loss function can also be used to calculate the loss value. For the aforementioned first matrix, second matrix and third matrix, the corresponding first label matrix, second label matrix and third label matrix can be preset respectively, and the diagonal elements of any label matrix are positive class labels, and the value is set to 1; the off-diagonal elements are negative class labels, and the value is set to 0. The first label matrix, the second label matrix and the third label matrix are concatenated to obtain a total label matrix. Based on the total label matrix and the aforementioned fourth matrix, the existing cross entropy loss function can be used to calculate the loss value, and the parameters of the multimodal encoding neural network can be updated based on the loss value.
[0059] According to an example of the present invention, the text entity "Leaning Tower of Pisa" is obtained, and the text entity is input into the text encoding model to obtain a multi-dimensional text entity vector (T1, T2, T3, ..., T n ); obtain the image entity (the picture of the Leaning Tower of Pisa), input the image entity into the image coding model to obtain a multi-dimensional image entity vector (I1, I2, I3, ..., I n ) ; Perform modal comparison learning on the multi-dimensional text entity vector and the so-called image entity vector to obtain a <T i , I i >The matrix of entity pairs, calculates the loss value through the loss function and reversely updates the parameters of the multimodal encoding neural network so that the text encoding model and the image encoding model can be trained.
[0060] According to one embodiment of the present invention, when training a text encoding model and an image encoding model, the loss value is configured to be negatively correlated with the distance between the vectors of different first samples and the distance between the vectors of different second samples, and the loss value is configured to be positively correlated with the distance between the vector of the first sample and the vector of the second sample corresponding to the relevant entity. For homomodal contrast learning, taking homomodal contrast learning between first samples as an example, because different first samples represent different entities, the distance between the vectors of different entities should be pulled away, and the greater the distance, the smaller the loss should be, so the loss value is configured to be negatively correlated with the distance between the vectors of different first samples, and the homomodal contrast learning between second samples is similar, which will not be repeated here; for cross-modal contrast learning, since the first sample and the second sample corresponding to the relevant entity are the same entity, the distance between the vectors of the same entity should be pulled closer, and the greater the distance, the greater the loss should be, so the loss value is configured to be positively correlated with the distance between the vector of the first sample and the vector of the second sample corresponding to the relevant entity. Thus, through this configuration, homomodal and cross-modal contrast learning can be efficiently realized, allowing the model to better learn the vector representation between homomodal and cross-modal entities.
[0061] 4. Build an index library
[0062] An index library storing multiple entities is constructed based on the above-mentioned trained coding neural network. The multidimensional in-library entity vectors of each in-library entity stored in the index library include multidimensional text entity vectors in a text entity index library constructed based on a text coding model and multidimensional image entity vectors in an image entity index library constructed based on an image coding model.
[0063] According to one embodiment of the present invention, a plurality of text entities are obtained, the plurality of text entities are encoded using a trained text encoding model to obtain a multidimensional vector, and the multidimensional vector is normalized to obtain a multidimensional text entity vector, the plurality of text entities and the multidimensional text entity vectors corresponding to the plurality of text entities are stored through an index library construction module, and a text entity index library including the plurality of text entities and the multidimensional text entity vectors corresponding to the plurality of text entities is constructed based on the plurality of text entities and the multidimensional text entity vectors corresponding to the plurality of text entities.
[0064] According to one embodiment of the present invention, a plurality of image entities are obtained, the plurality of image entities are encoded using a trained image coding model to obtain a multidimensional vector, and the multidimensional vector is normalized to obtain a multidimensional image entity vector, the plurality of image entities and the multidimensional image entity vectors corresponding to the plurality of image entities are stored through an index library construction module, and an image entity index library including the plurality of image entities and the multidimensional image entity vectors corresponding to the plurality of image entities is constructed based on the plurality of image entities and the multidimensional image entity vectors corresponding to the plurality of image entities.
[0065] 5. Application Scenarios
[0066] The present invention recalls the query entity based on the above-mentioned model structure, training samples, training process and index library construction, see Figure 2 , including steps T1, T2 and T3, each of which is described in detail below.
[0067] Step T1: Obtain an entity to be queried, and use a trained multimodal encoding neural network and a preset normalization method to extract a multidimensional entity vector of the entity to be queried, wherein the multimodal encoding neural network includes a text encoding model for extracting vectors for input entities in text modality and an image encoding model for extracting vectors for input entities in image modality, and is obtained based on same-modality contrast learning and cross-modality contrast learning.
[0068] According to one embodiment of the present invention, the trained encoding neural network is obtained during the third part of the training process; the preset normalization method is the L2 normalization method in the first part.
[0069] According to one embodiment of the present invention, the entity to be queried may be a text entity vector. When the entity to be queried is a text entity vector, a trained text encoding model is used to encode the entity to be queried to obtain a multi-dimensional vector and the multi-dimensional vector is L2 normalized to obtain a multi-dimensional entity vector of the entity to be queried.
[0070] According to one embodiment of the present invention, the entity to be queried may also be an image entity vector. When the entity to be queried is an image entity vector, a trained image coding model is used to encode the entity to be queried to obtain a multi-dimensional vector and the multi-dimensional vector is L2 normalized to obtain a multi-dimensional entity vector of the entity to be queried.
[0071] Step T2: Calculate the similarity between the multidimensional entity vector to be queried and the multidimensional in-library entity vectors of each in-library entity stored in the index library, wherein the multidimensional in-library entity vectors of each in-library entity are extracted in advance based on the trained multimodal encoding neural network and a preset normalization method.
[0072] In this step, based on the multidimensional entity vector of the entity to be queried obtained in step T1, the similarity calculation module calculates the similarity between the multidimensional entity vector and the multidimensional entity vectors of each entity in the index library.
[0073] According to one embodiment of the present invention, the index library is a text entity index library constructed in the fourth part, which includes multiple text entities and multidimensional text entity vectors corresponding to the multiple text entities, and an image entity index library which includes multiple image entities and multidimensional image entity vectors corresponding to the multiple image entities.
[0074] According to one embodiment of the present invention, when the entity to be queried is a text entity, the similarity between the multidimensional entity vector of the entity to be queried and the multidimensional in-library entity vectors of each in-library entity stored in the image entity index library is calculated; when the entity to be queried is an image entity, the similarity between the multidimensional entity vector of the entity to be queried and the multidimensional in-library entity vectors of each in-library entity stored in the text entity index library is calculated.
[0075] Step T3: Based on the similarity, multiple entities in the index library are recalled as candidate entities.
[0076] In this step, based on the similarity between the multi-dimensional entity vector of the entity to be queried calculated in step T2 and the multi-dimensional entity vector of each entity in the index library stored in the index library, multiple entities in the index library are recalled as candidate entities. According to one embodiment of the present invention, all entities in the index library are sorted in descending order based on the calculated similarity by the entity recall module, and the top K entities in the index library are recalled as candidate entities.
[0077] According to one embodiment of the present invention, when the entity to be queried is text, multiple image entities in the image entity index library are recalled as candidate entities. For example, when the entity to be queried is "Eiffel Tower", the recalled image entities are multiple images of "Eiffel Tower".
[0078] According to one embodiment of the present invention, when the entity to be queried is an image, multiple text entities in the text entity index library are recalled as candidate entities. For example, when the entity to be queried is a picture of a cat, the multiple text entities recalled may be "cat", "kitty", "Maine Coon cat", etc.
[0079] It should be noted that, in some embodiments of the present invention, when the entity to be queried is text, the multiple recalled in-library entities also include text entities in the text entity index library. For example, when the entity to be queried is "loss function", the multiple recalled in-library entities also include text entities "cross entropy loss function", "absolute value loss function", "Loss Function" and the like in the text entity index library; when the entity to be queried is an image, the multiple recalled in-library entities also include image entities in the image entity index library. For example, when the entity to be queried is a picture of a cat, the multiple recalled in-library entities also include images of cats in the image entity index library.
[0080] According to one embodiment of the present invention, there is provided an entity recall device, comprising:
[0081] A text encoding module, used for extracting a multi-dimensional entity vector of a to-be-queried entity from an entity in a text mode by using a text encoding model and a preset normalization method;
[0082] An image coding module, used for extracting a multi-dimensional entity vector of the entity to be queried from an entity to be queried in an image mode by using an image coding model and a preset normalization method;
[0083] An index library construction module, used to construct a text entity index library storing multiple multi-dimensional text entity vectors using a text encoding module and to construct an image entity index library storing multiple multi-dimensional image entity vectors using an image encoding module;
[0084] A similarity calculation module is used to calculate the similarity between the multidimensional entity vector to be queried and the multidimensional in-library entity vectors of each in-library entity stored in the index library;
[0085] The entity recall module is used to recall multiple entities in the index library as candidate entities based on the similarity calculated by the similarity calculation module.
[0086] In summary, the present invention proposes a cross-modal entity recall method based on modality contrast learning. Compared with the traditional entity mask reconstruction task, the present invention not only introduces contrast learning within the modality and contrast learning between modalities, which shortens the distance between related entity pairs between modalities in space, but also increases the distance between different entities within the modality in space, ensuring the uniform distribution of entities in space, thereby realizing rapid retrieval of entities; the present invention also calculates entity vectors in advance based on the dual-tower model combined with the common vector recall model and stores them offline, so that during online retrieval, only the similarity between the entity to be queried and the entity vector in the index library needs to be calculated to quickly recall the entity, reducing space consumption.
[0087] It should be noted that although the above describes the various steps in a specific order, it does not mean that the various steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.
[0088] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.
[0089] Computer readable storage medium can be a tangible device that holds and stores instructions used by an instruction execution device. Computer readable storage medium can include, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a protruding structure in a groove on which instructions are stored, and any suitable combination thereof.
[0090] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.
Claims
1. A cross-modal entity recall method based on modality contrast learning, characterized in that: The method comprises the following steps: T1. Obtain an entity to be queried, and extract a multi-dimensional entity vector of the entity to be queried using a trained multimodal encoding neural network and a preset normalization method, wherein the multimodal encoding neural network includes a text encoding model for extracting vectors from an input entity in a textual modality and an image encoding model for extracting vectors from an input entity in an image modality, and is obtained based on same-modality contrast learning and cross-modality contrast learning; T2. Calculate the similarity between the multidimensional entity vector to be queried and the multidimensional in-library entity vectors of each in-library entity stored in the index library, wherein the multidimensional in-library entity vectors of each in-library entity are extracted in advance based on the trained multimodal encoding neural network and a preset normalization method; T3. Based on the similarity, recall multiple entities in the index library as candidate entities.
2. The method according to claim 1, characterized in that The same-modality contrastive learning is used to increase the distance between vectors output by encoding models of the corresponding modality for different entities, and the cross-modality contrastive learning is used to decrease the distance between vectors output by encoding models of different modalities for related entities.
3. The method according to claim 2, characterized in that The trained multimodal encoding neural network is trained in the following manner: Acquire multiple first samples of text modality and multiple second samples of image modality, each first sample is provided with a second sample of different modality but corresponding to an entity, and establish an association relationship between the two, different first samples correspond to different entities, and different second samples correspond to different entities; Inputting the first sample into a text encoding model to obtain a vector of the first sample; Inputting the second sample into an image coding model to obtain a vector of the second sample; The same-modal comparative learning is performed using the distance between the vectors of different first samples and the distance between the vectors of different second samples. Based on the association relationship, the distance between the vector of the first sample and the vector of the second sample corresponding to the related entity is used to perform cross-modal comparative learning.
4. The method according to claim 3, characterized in that During training, the loss value is configured to be negatively correlated with the distance between the vectors of different first samples and the distance between the vectors of different second samples, and the loss value is configured to be positively correlated with the distance between the vector of the first sample and the vector of the second sample corresponding to the related entity.
5. The method according to claim 4, characterized in that Homomodal contrastive learning constructs a first matrix for multiple first samples and a second matrix for multiple second samples, wherein the diagonal elements of the first matrix and the second matrix represent similarities determined based on vectors of homomodal positive samples, and the off-diagonal elements represent similarities determined based on vectors of homomodal negative samples; The cross-modal contrastive learning constructs a third matrix for the first sample and the second sample corresponding to the relevant entity, wherein the diagonal elements of the third matrix represent the similarity of the cross-modal positive samples, and the non-diagonal elements represent the similarity determined based on the vector of the cross-modal negative samples; The loss value is calculated as follows to update the parameters of the multimodal encoding neural network: in, represents the similarity of a single cross-modal positive sample, τ represents the preset hyperparameter, exp(·) represents the mean value, Represents the similarity of a single homomodal negative sample or cross-modal negative sample, k∈n, j∈n.
6. The method according to claim 1, characterized in that In step T1, If the entity to be queried is a text, the text encoding model is used to encode the entity to be queried to obtain a multi-dimensional vector and the multi-dimensional vector is L2 normalized to obtain a multi-dimensional entity vector of the entity to be queried; If the entity to be queried is an image, the image coding model is used to encode the entity to be queried to obtain a multi-dimensional vector and the multi-dimensional vector is L2 normalized to obtain a multi-dimensional entity vector of the entity to be queried.
7. The method according to claim 6, characterized in that In step T2, the index library includes a text entity index library constructed based on a text encoding model and an image entity index library constructed based on an image encoding model, and the calculating of the similarity between the multidimensional entity vector to be queried and the multidimensional in-library entity vectors of each in-library entity stored in the index library includes: If the entity to be queried is text, calculating the similarity between the multi-dimensional entity vector of the entity to be queried and the multi-dimensional entity vector of each entity in the library stored in the image entity index library; If the entity to be queried is an image, the similarity between the multi-dimensional entity vector of the entity to be queried and the multi-dimensional in-library entity vectors of each in-library entity stored in the text entity index library is calculated.
8. An entity recall device based on the method according to any one of claims 1 to 7, characterized in that: The device comprises: A text encoding module, used for extracting a multi-dimensional entity vector of a to-be-queried entity from an entity in a text mode by using a text encoding model and a preset normalization method; An image coding module, used for extracting a multi-dimensional entity vector of the entity to be queried from an entity to be queried in an image mode by using an image coding model and a preset normalization method; An index library construction module, used to construct a text entity index library storing multiple multi-dimensional text entity vectors using a text encoding module and to construct an image entity index library storing multiple multi-dimensional image entity vectors using an image encoding module; A similarity calculation module is used to calculate the similarity between the multidimensional entity vector to be queried and the multidimensional in-library entity vectors of each in-library entity stored in the index library; The entity recall module is used to recall multiple entities in the index library as candidate entities based on the similarity calculated by the similarity calculation module.
9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of any method described in claims 1 to 7.
10. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to implement the steps of the method as claimed in any one of claims 1 to 7.
Citation Information
Patent Citations
Five-mode commodity pre-training method and retrieval system based on self-coordination comparative learning
CN114418032A
Visual-semantic representation learning via multi-modal contrastive training
US20220284321A1
