Cross-modal retrieval method, training method of cross-modal retrieval model, and related device

Abstract

The application discloses a cross-modal retrieval method, a training method of a cross-modal retrieval model, an electronic device and a computer readable storage medium. The cross-modal retrieval method first extracts a feature vector of a first modal sample as a retrieval target to generate a first unit vector set, and extracts feature vectors of multiple second modal samples as candidate objects to generate a second unit vector set. Then, the first unit vector set and the second unit vector set are respectively subjected to pooling processing. The first global vector and the multiple second global vectors obtained through the pooling are respectively projected into the same joint space. Finally, a cross-modal retrieval result is obtained based on the similarity between the first joint vector and each second joint vector obtained through the projection. The first global vector contains position information of a first unit contained in the first modal sample, and the second global vector contains position information of a second unit contained in the second modal sample, so that the accuracy of the retrieval result can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of retrieval, in particular to a cross-modal retrieval method, a training method of a cross-modal retrieval model, an electronic device and a computer readable storage medium. BACKGROUND

[0002] In the traditional cross-modal retrieval technology, it is usually necessary to project the features of samples of different modalities into the same embedding space to obtain a retrieval result. In the process of obtaining the features, the sample is usually divided into multiple units, and then the features of each unit are extracted, and the features of all units are pooled to obtain the features of the sample. The commonly used pooling is average pooling, but average pooling is easy to cause the loss of important information of the sample, thereby reducing the accuracy of the retrieval result. SUMMARY

[0003] The technical problem solved by the present application is to provide a cross-modal retrieval method, a training method of a cross-modal retrieval model, an electronic device and a computer readable storage medium, which can improve the accuracy of the retrieval result.

[0004] To solve the above technical problem, one technical solution adopted by the present application is to provide a cross-modal retrieval method, comprising:

[0005] inputting a first modality sample as a retrieval target and a plurality of second modality samples as candidate objects into a retrieval model, wherein the retrieval model comprises a first feature extraction network, a second feature extraction network, a pooling network, a first embedding network and a second embedding network;

[0006] extracting a first unit vector set of the first modality sample by using the first feature extraction network, and extracting a second unit vector set of each second modality sample by using the second feature extraction network, the first modality sample comprising a plurality of first units, the first unit vector set being composed of feature vectors of all first units, each second modality sample comprising a plurality of second units, and each second unit vector set being composed of feature vectors of all second units in the corresponding second modality sample;

[0007] pooling the first unit vector set by using the pooling network to obtain a first global vector, and respectively pooling each second unit vector set by using the pooling network to obtain a plurality of second global vectors, wherein the first global vector contains position information of the first unit in the first modality sample, and each second global vector contains corresponding position information of the second unit in the corresponding second modality sample;

[0008] projecting the first global vector into a joint space by using the first embedding network to obtain a first joint vector, and respectively projecting each second global vector into the joint space by using the second embedding network to obtain a plurality of second joint vectors;

[0009] obtaining a cross-modal retrieval result of the first modality sample based on the similarity between the first joint vector and each second joint vector.

[0010] To solve the above technical problems, another technical solution adopted by the present application is to provide a training method of a cross-modal retrieval model, comprising:

[0011] inputting a plurality of first modality samples and a plurality of second modality samples into a retrieval model, wherein the retrieval model comprises a first feature extraction network, a second feature extraction network, a pooling network, a first embedding network and a second embedding network, each first modality sample and / or each second modality sample has labeled data;

[0012] extracting a first unit vector set of each first modality sample by using the first feature extraction network, and extracting a second unit vector set of each second modality sample by using the second feature extraction network, each first modality sample comprising a plurality of first units, each first unit vector set being composed of feature vectors of all first units in the corresponding first modality sample, each second modality sample comprising a plurality of second units, and each second unit vector set being composed of feature vectors of all second units in the corresponding second modality sample;

[0013] pooling each first unit vector set by using the pooling network to obtain a plurality of first global vectors, and pooling each second unit vector set by using the pooling network to obtain a plurality of second global vectors, wherein each first global vector contains position information of the first unit in the corresponding first modality sample, and each second global vector contains position information of the second unit in the corresponding second modality sample;

[0014] projecting each first global vector into a joint space by using the first embedding network to obtain a plurality of first joint vectors, and projecting each second global vector into the joint space by using the second embedding network to obtain a plurality of second joint vectors;

[0015] adjusting parameters of the retrieval model based on the first joint vector and the second joint vector.

[0016] To solve the above technical problems, another technical solution adopted by the present application is to provide an electronic device, comprising a memory and a processor coupled to each other, the memory storing program instructions, and the processor being capable of executing the program instructions to implement the cross-modal retrieval method of the above technical solution or the training method of the cross-modal retrieval model of the above technical solution.

[0017] To solve the above technical problems, another technical solution adopted by the present application is to provide a computer readable storage medium, the storage medium stores program instructions, the program instructions can be executed by the processor to implement the cross-modal retrieval method of the above technical solution, or the training method of the cross-modal retrieval model of the above technical solution.

[0018] To solve the above technical problems, another technical solution adopted by the present application is to provide an electronic device, characterized in that the electronic device comprises a memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor can execute the program instructions to implement:

[0019] Input the first modal sample as the retrieval target and the plurality of second modal samples as the candidate objects into the retrieval model, wherein the retrieval model comprises a first feature extraction network, a second feature extraction network, a pooling network, a first embedding network and a second embedding network;

[0020] Extract a first unit vector set of the first modal sample by using the first feature extraction network, and extract a second unit vector set of each second modal sample by using the second feature extraction network, the first modal sample comprises a plurality of first units, the first unit vector set is composed of feature vectors of all first units, and each second modal sample comprises a plurality of second units, and each second unit vector set is composed of feature vectors of all second units in the corresponding second modal sample;

[0021] Pool the first unit vector set by using the pooling network to obtain a first global vector, and pool each second unit vector set by using the pooling network to obtain a plurality of second global vectors, wherein the first global vector contains position information of the first unit in the first modal sample, and each second global vector contains corresponding position information of the second unit in the corresponding second modal sample;

[0022] Project the first global vector into a joint space by using the first embedding network to obtain a first joint vector, and project each second global vector into the joint space by using the second embedding network to obtain a plurality of second joint vectors;

[0023] Obtain the cross-modal retrieval result of the first modal sample based on the similarity between the first joint vector and each second joint vector.

[0024] The application has the beneficial effects that: in the cross-modal retrieval method provided by the application, the first feature extraction network can extract a first unit vector set of the first modal sample, the second feature extraction network can extract a second unit vector set of each second modal sample respectively, the first modal sample can include a plurality of first units, the first unit vector set is composed of feature vectors of all first units, each second modal sample can include a plurality of second units, each second unit vector set is composed of feature vectors of all second units of the corresponding second modal sample, the pooling network can pool the first unit vector set to obtain a first global vector, and pool each second unit vector set respectively to obtain a plurality of second global vectors, wherein the first global vector contains corresponding position information of the first unit in the first modal sample, and each second global vector contains corresponding position information of the second unit in the corresponding second modal sample, the first embedding network can project the first global vector to a joint space to obtain a first joint vector, the second embedding network can project each second global vector to the joint space respectively to obtain a plurality of second joint vectors, and then the cross-modal retrieval result of the first modal sample can be obtained based on the similarity between the first joint vector and each second joint vector. That is, the application retains the position information in different modal samples in the pooling process, thereby improving the accuracy of the retrieval result. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0026] Figure 1 Flow chart of an embodiment of the cross-modal retrieval method of the application;

[0027] Figure 2a One of the retrieval result examples of image matching;

[0028] Figure 2b The second of the retrieval result examples of image matching;

[0029] Figure 2c The third of the retrieval result examples of image matching;

[0030] Figure 3 Flow chart of an embodiment of step S13 in the method; Figure 1

[0031] Flow chart of an embodiment of step S21 in the method; Figure 4 Figure 3

[0032] ​​Figure 5 As Figure 3 A flowchart of an embodiment of step S22;

[0033] Figure 6 As Figure 1 A flowchart of another embodiment of step S13;

[0034] Figure 7 As Figure 6 A flowchart of an embodiment of step S51;

[0035] Figure 8 As Figure 6 A flowchart of an embodiment of step S52;

[0036] Figure 9 A structural diagram of an embodiment of the cross-modal retrieval device of the present application;

[0037] Figure 10 A flowchart of an embodiment of the training method of the cross-modal retrieval model of the present application;

[0038] Figure 11 As Figure 10 A flowchart of an embodiment of step S85;

[0039] Figure 12 A structural diagram of an embodiment of the electronic device of the present application;

[0040] Figure 13 A structural diagram of an embodiment of the computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0041] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0042] Please refer to Figure 1 , Figure 1 A flowchart of an embodiment of the cross-modal retrieval method of the present application, which comprises the following steps:

[0043] Step S11, inputting a first modal sample as a retrieval target and a plurality of second modal samples as candidate objects into a retrieval model, wherein the retrieval model comprises a first feature extraction network, a second feature extraction network, a pooling network, a first embedding network and a second embedding network.

[0044] In the cross-modal retrieval process, a first modality sample is taken as a retrieval target, and multiple second modality samples are taken as candidate objects, for example, text samples, image samples, audio samples, and other samples of different modalities. The purpose of retrieval is to obtain several second modality samples matching the first modality sample. For ease of understanding, the present embodiment takes the first modality sample as a text sample and the second modality sample as an image sample for illustration. For example, the present embodiment first inputs a text sample and multiple image samples into a retrieval model to obtain one or more image samples matching the text sample as image matching retrieval results. The retrieval model includes a first feature extraction network, a second feature extraction network, a pooling network, a first embedding network, and a second embedding network, and the retrieval model is pre-trained. The specific training method will be described later.

[0045] In step S12, the first feature extraction network is used to extract a first unit vector set of the first modality sample, and the second feature extraction network is used to extract a second unit vector set of each second modality sample. The first modality sample includes multiple first units, and the first unit vector set is composed of feature vectors of all first units. Each second modality sample includes multiple second units, and each second unit vector set is composed of feature vectors of all second units in the corresponding second modality sample.

[0046] The first modality sample includes multiple first units, and the second modality sample includes multiple second units. The present embodiment can use the first feature extraction network to extract the feature vector of each first unit to obtain the first unit vector set, and can use the second feature extraction network to extract the feature vector of each second unit to obtain the second unit vector set. The second unit vector set corresponds to the second modality sample one by one.

[0047] Taking a text sample as the first modality sample and an image sample as the second modality sample as an example, the text sample includes multiple words, and the image sample includes multiple salient regions, each of which contains a target of interest. The first feature extraction network can extract multiple word vectors in the text sample, and the second feature extraction network can extract multiple region vectors in the image sample, thereby obtaining a word vector set corresponding to the text sample and a region vector set corresponding to the image sample, and each image sample corresponds to a region vector set. The first feature extraction network includes but is not limited to a skip-gram model, which can vectorize all word vectors of the text sample. The second feature extraction network includes but is not limited to a Faster R-CNN model and a ResNet model, which can detect the target of interest from the image sample and vectorize the salient region containing the target of interest.

[0048] Step S13, the first global vector is obtained by pooling the first cell vector set using a pooling network, and the plurality of second global vectors are obtained by respectively pooling each second cell vector set using a pooling network, wherein the first global vector contains the position information of the first cell in the first modality sample, and each second global vector contains the corresponding position information of the second cell in the corresponding second modality sample.

[0049] Each first cell in the first modality sample has its own absolute position information and relative position information with other first cells, and each second cell in the second modality sample also has its own absolute position information and relative position information with other second cells. The embodiment obtains the first global vector and the second global vector by respectively pooling the first cell vector set and the second cell vector set, and preserving the position information in the pooling process, wherein the first global vector corresponds to the first modality sample as the retrieval target, and each second global vector corresponds to one of the plurality of second modality samples as the candidate object.

[0050] Still taking the text sample as the first modality sample and the image sample as the second modality sample as an example, each word in the text sample has absolute position information in the text sample and relative position information with other words, and each salient region in the image sample has absolute position information in the image sample and relative position information with other salient regions. The embodiment vectorizes the position information of each word and the position information of each salient region, and adds the respective corresponding feature vectors, so as to preserve the position information in the pooling process. That is, the pooling process in the embodiment is "sequential encoding hybrid pooling". After pooling, the first global vector of the text sample and the second global vector of each image sample are obtained. The specific pooling process will be described later.

[0051] Step S14, the first global vector is projected into the joint space to obtain the first joint vector using the first embedding network, and each second global vector is respectively projected into the joint space to obtain the plurality of second joint vectors using the second embedding network.

[0052] After obtaining the first global vector and the plurality of second global vectors respectively, it is necessary to project them into the same joint space by using embedding networks, and then retrieval can be performed. Specifically, the first embedding network is used to project the first global vector into the joint space to obtain the first joint vector, and the second embedding network is used to project each second global vector into the joint space to obtain a plurality of second joint vectors. That is, the first joint vector corresponds to the first modality sample as the retrieval target, and each second joint vector corresponds to one of the plurality of second modality samples as the candidate object, so that the relationship between the retrieval target and each candidate object can be obtained by comparing the relationship between the first joint vector and each second joint vector.

[0053] For example, the first global vector corresponding to the text sample and the second global vector corresponding to each of the plurality of image samples are projected into the same image semantic joint space, so that the relationship comparison between the text sample and each image sample can be obtained.

[0054] In step S15, the cross-modal retrieval result of the first modality sample is obtained based on the similarity between the first joint vector and each second joint vector.

[0055] After obtaining the first joint vector and the plurality of second joint vectors, the embodiment obtains the cross-modal retrieval result by obtaining the similarity between the first joint vector and each second joint vector. That is, the similarity between the first joint vector and each second joint vector can be used to quantitatively compare the relationship between the corresponding first modality sample and each second modality sample, so as to obtain the retrieval result.

[0056] Specifically, the second modality samples corresponding to the specified number of second joint vectors with the highest similarity can be selected as the cross-modal retrieval result of the first modality sample. That is, the distance between the first joint vector and each second joint vector in the joint space can be calculated, and the distance is sorted to output the second modality samples corresponding to the specified number (for example, 8, 10, 15, etc.) of second joint vectors with the closest distance as the retrieval result.

[0057] For example, the similarity between the text sample and each image sample can be obtained and sorted, so as to output the image sample with the highest similarity as the retrieval result of image matching. Please refer to Figure 2a 、 Figure 2b and Figure 2c , which are retrieval result examples of image matching, wherein, Figure 2a is the retrieval result of image matching of the text sample "a dog playing on the grass", Figure 2b is the retrieval result of image matching of the text sample "riding a motor bike",Figure 2c For the retrieval result of image matching of the text sample "smiling people", the top 10 image samples with the highest similarity are output. It can be seen that the embodiment improves the accuracy of the retrieval result by preserving the position information in different modal samples in the pooling process.

[0058] In one embodiment, please refer to Figure 3 , Figure 3 For Figure 1 An embodiment of the flowchart of step S13 in the foregoing embodiment, the first global vector can be obtained by the following steps.

[0059] Step S21, the position information of each first unit in the first modal sample is embedded into the corresponding feature vector to obtain a first sequential coding vector set.

[0060] The embodiment can preserve the position information of each first unit in the first modal sample in the process of obtaining the first global vector, specifically, the position information is embedded into the corresponding feature vector to obtain a first sequential coding vector set.

[0061] In one embodiment, please refer to Figure 4 , Figure 4 For Figure 3 An embodiment of the flowchart of step S21 in the foregoing embodiment, the position information can be embedded into the corresponding feature vector by the following steps.

[0062] Step S31, a first position vector of each first unit is generated based on the position information of each first unit in the first modal sample, and the first position vector has the same dimension as the corresponding feature vector.

[0063] Taking a text sample as the first modal sample, a first position vector of each word is generated based on the position information of each word in the text sample, that is, the position information is vectorized, and the first position vector has the same dimension as the corresponding feature vector, which is convenient for embedding the feature vector.

[0064] Specifically, the first position vector can be obtained by the following formula (1):

[0065]

[0066] wherein PE represents the first position vector, pos represents the position of the first unit in the first modal sample, D represents the dimension of the first position vector PE, i represents an integer between 0 and (D-1) / 2, such as 0, 1, 2, …, 2i represents an even dimension in the dimension of the first position vector PE, for example, the zero-th dimension, the second dimension, the fourth dimension, …, 2i+1 represents an odd dimension in the dimension of the first position vector PE, for example, the first dimension, the third dimension, the fifth dimension, …, PE(pos, 2i) represents the value of the position pos and the 2i-th dimension, which can be calculated by a sine function, and PE(pos, 2i+1) represents the value of the position pos and the 2i+1-th dimension, which can be calculated by a cosine function.

[0067] In step S32, the feature vector of each first unit in the first unit vector set is replaced by the corresponding first order encoding vector, and a first order encoding vector set is obtained, wherein the first order encoding vector of each first unit is the sum of the first position vector and the feature vector of the first unit.

[0068] After obtaining the first position vector, the first order encoding vector can be obtained by adding the corresponding feature vector, and thus the first order encoding vector set is obtained. That is, the first order encoding vector set can be obtained by the following formula (2):

[0069] X' = X + PE …… (2);

[0070] wherein X represents the feature vector of the first unit, and X' represents the corresponding first order encoding vector. For example, in a text sample, X is the feature vector corresponding to a word in the text sample, and X' is the first order encoding vector of the word.

[0071] The embodiment can obtain the first position vector by vectorizing the position information of the first unit, and take the sum of the first position vector and the corresponding feature vector as the first order encoding vector, thereby retaining the position information of the first unit in the first modal sample and improving the accuracy of the retrieval result.

[0072] In step S22, the first order encoding vector set is pooled by using a pooling network to obtain a first global vector.

[0073] The pooling network includes a first full connection layer, a second full connection layer and a third full connection layer, and the first order encoding vector set is pooled by using the three full connection layers, which will be described in detail in Figure 5 , Figure 5 For Figure 3 The flowchart of an embodiment of step S22 in the method is shown in FIG. 4, and the first order encoding vector set can be pooled by the following steps.

[0074] In step S41, the first order encoding vector set is clustered based on a soft assignment function to obtain a plurality of classes.

[0075] The clustering operation can place several first-order encoding vectors closest in distance into a class, and the index s of each first-order encoding vector in each class can be obtained by a soft assignment function shown in the following formula (3) k (x′ i ), to realize the clustering operation:

[0076]

[0077] wherein K is the number of classes, x′ i is the first-order encoding vector in the kth class, and b k are learnable parameters related to the kth class, k is an integer between 1 and K, and s k (x′ i ) represents the probability that the first-order encoding vector x′ i belongs to the kth class.

[0078] In step S42, the first pooling feature of each class is obtained based on the first full connection layer, and the second pooling feature of each class is obtained based on the second full connection layer.

[0079] After the clustering operation, the first pooling feature G1(X′,k) of each class can be obtained based on the first full connection layer , and the second pooling feature G2(X′,k) of each class can be obtained based on the second full connection layer , specifically, the first pooling feature and the second pooling feature of each class are obtained by the following formula (4) respectively:

[0080]

[0081] wherein N is the number of first-order encoding vectors x′ i in the kth class, c k is the clustering center of the kth class, and σ k is the normalization parameter of the kth class.

[0082] In step S43, the sum of the first pooling features and the second pooling features of all classes is input into the third full connection layer to obtain the first global vector.

[0083] After the first pooling feature G1(X′,k) and the second pooling feature G2(X′,k) of each class are obtained respectively, the sum of the first pooling features of all classes and the sum of the second pooling features of all classes are calculated, and the sum of the two is input into the third full connection layer Input the sum of the first pooled features and the second pooled features of all classes into a third fully connected layer to obtain a first global vector P OEM (X′). The process can be represented by the following formula (5):

[0084]

[0085] The embodiment pools the first sequential encoding vector sets containing position information through multiple fully connected layers to obtain the first global vector of the first modal sample, thereby improving the accuracy of the retrieval result.

[0086] In one embodiment, refer to Figure 6 , Figure 6 To Figure 1 In another embodiment of step S13 in the method, the second global vector can be obtained through the following steps.

[0087] In step S51, the position information of each second unit in each second modal sample is embedded into the corresponding feature vector respectively to obtain multiple second sequential encoding vector sets.

[0088] The embodiment can retain the position information of the second unit in the second modal sample in the process of obtaining the second global vector, specifically, the position information of each second unit in each second modal sample is embedded into the corresponding feature vector respectively to obtain multiple second sequential encoding vector sets.

[0089] In one embodiment, refer to Figure 7 , Figure 7 To Figure 6 In one embodiment of step S51 in the method, the position information can be embedded into the corresponding feature vector through the following steps.

[0090] In step S61, a second position vector of each second unit is generated based on the position information of each second unit in the corresponding second modal sample, and the second position vector has the same dimension as the corresponding feature vector.

[0091] Taking an image sample as the second modal sample, a second position vector of each salient region is generated based on the position information of each salient region in the image sample, that is, the position information is vectorized, and the second position vector has the same dimension as the corresponding feature vector, facilitating embedding into the feature vector.

[0092] Since the second position vector contains the row parameter and the column parameter of the salient region, half of the elements in the second position vector can be used to encode the row parameter, and the other half of the elements can be used to encode the column parameter. Among them, the elements with even serial numbers in the second position vector conform to the sine rule, and the elements with odd serial numbers conform to the cosine rule. The specific calculation process of the second position vector is the same as that of the first position vector described above, and can be referred to the above formula (1).

[0093] Step S62, replace the feature vector of each second unit in each second unit vector set with the corresponding second sequential encoding vector to obtain a plurality of second sequential encoding vector sets, each second sequential encoding vector being the sum of the second position vector and the feature vector of the corresponding second unit.

[0094] After obtaining the second position vector of each second unit in each second unit vector set respectively, the second position vector can be added to the corresponding feature vector to obtain a second sequential encoding vector, thereby obtaining a plurality of second sequential encoding vector sets. For details, refer to formula (2) above.

[0095] That is, replace PE in formula (1) above with the second position vector, replace pos with the position of the second unit in the second modality sample, and replace (D+1) with the dimension of the second position vector PE. Replace X in formula (2) above with the feature vector of the second unit, and replace X' with the corresponding second sequential encoding vector. Then, use formula (1) and formula (2) to obtain a plurality of second sequential encoding vector sets.

[0096] The embodiment can obtain the second position vector by vectorizing the position information of the second unit, and use the sum of the second position vector and the corresponding feature vector as the second sequential encoding vector. The position information of the second unit in the second modality sample is retained, and the accuracy of the retrieval result can be improved.

[0097] Step S52, pool each second sequential encoding vector set using a pooling network to obtain a plurality of second global vectors.

[0098] The pooling network includes a first fully connected layer, a second fully connected layer, and a third fully connected layer. The embodiment uses the three fully connected layers to pool each second sequential encoding vector set. For details, refer to Figure 8 , Figure 8 For Figure 6 An embodiment of the flowchart of step S52 is shown in the figure. The second sequential encoding vector set can be pooled by the following steps.

[0099] Step S71, cluster each second sequential encoding vector set based on a soft assignment function to obtain a plurality of classes.

[0100] The clustering operation can place several second sequential encoding vectors that are closest in distance in a class. For details, refer to formula (3) above. It should be noted that there are multiple second sequential encoding vector sets, and the clustering operation can be performed on each second sequential encoding vector set to obtain a plurality of classes corresponding to each second sequential encoding vector set.

[0101] Step S72, based on the first full connection layer, respectively obtaining the first pooling features of each class corresponding to each second order encoding vector set, and based on the second full connection layer, respectively obtaining the second pooling features of each class corresponding to each second order encoding vector set.

[0102] After the clustering operation, for each second order encoding vector set, the first pooling features of each class can be obtained based on the first full connection layer, and the second pooling features of each class can be obtained based on the second full connection layer, and the specific process can be referred to the above formula (4).

[0103] Step S73, inputting the sum of the first pooling features and the second pooling features of all classes corresponding to each second order encoding vector set into the third full connection layer respectively, to obtain a plurality of second global vectors.

[0104] After obtaining the first pooling features and the second pooling features of each class respectively for each second order encoding vector set, the sum of the first pooling features of all classes and the sum of the second pooling features of all classes are calculated first, and then the sum of the two is input into the third full connection layer, that is, the sum of the first pooling features and the second pooling features of all classes is input into the third full connection layer to obtain the second global vector corresponding to each second order encoding vector set, and the specific process can be referred to the above formula (5).

[0105] That is, K in the above formula (3) is represented as the number of classes corresponding to the second order encoding vector set, x′ i is represented as the second order encoding vector in the kth class corresponding to each second order encoding vector set, N in the above formula (4) is represented as the number of second order encoding vectors x′ i in the kth class corresponding to each second order encoding vector set, P OEM (X′) is represented as the second global vector corresponding to each second order encoding vector set, and then a plurality of second global vectors are obtained by using formula (3), formula (4) and formula (5).

[0106] The embodiment can pool the plurality of second order encoding vector sets containing position information through the plurality of full connection layers, thereby obtaining the second global vector corresponding to each second modal sample respectively, and the accuracy of the retrieval result can be improved.

[0107] Based on the same inventive concept, the application also provides a cross-modal retrieval device, please refer to Figure 9 , Figure 9As shown in FIG. 1, the structure of an embodiment of the cross-modal retrieval device of the present application includes a first feature extraction module 110, a second feature extraction module 120, a pooling module 130, a first embedding module 140, a second embedding module 150, and an obtaining module 160. The first modal sample as a retrieval target is the input of the first feature extraction module 110, and the plurality of second modal samples as candidate objects can be the input of the second feature extraction module 120. The outputs of the first feature extraction module 110 and the second feature extraction module 120 can be the input of the pooling module 130. The output of the pooling module 130 corresponding to the output of the first feature extraction module 110 can be the input of the first embedding module 140, and the output of the pooling module 130 corresponding to the output of the second feature extraction module 120 can be the input of the second embedding module 150. The outputs of the first embedding module 140 and the second embedding module 150 can be the input of the obtaining module 160, and the output of the obtaining module 160 can be the cross-modal retrieval result.

[0108] The first feature extraction module 110 is used to extract a first set of unit vectors of the first modal sample, and the second feature extraction module 120 is used to extract a second set of unit vectors of each second modal sample, respectively. The first modal sample includes a plurality of first units, and the first set of unit vectors is composed of feature vectors of all first units. The second modal sample includes a plurality of second units, and the second set of unit vectors is composed of feature vectors of all second units.

[0109] The pooling module 130 is used to pool the first set of unit vectors to obtain a first global vector, and is used to pool each second set of unit vectors to obtain a plurality of second global vectors, respectively. The first global vector contains the position information of the first unit in the first modal sample, and each second global vector contains the position information of the corresponding second unit in the second modal sample.

[0110] The pooling module 130 includes a sequential encoding module 131 and a fully connected module 132.

[0111] When the first unit vector set is input to the pooling module 130, the sequential encoding module 131 is configured to embed the position information of each first unit in the first modality sample into the corresponding feature vector, output a first sequential encoding vector set, and the full connection module 132 is configured to pool the first sequential encoding vector set, output a first global vector. Specifically, the sequential encoding module 131 first generates a first position vector of each first unit based on the position information of each first unit in the first modality sample, and then takes the sum of each first position vector and the corresponding feature vector as a first sequential encoding vector, thereby obtaining the first sequential encoding vector set. Specifically, the full connection module 132 first clusters the first sequential encoding vector set based on a soft assignment function to obtain multiple classes, then obtains the first pooling feature and the second pooling feature of each class respectively, and finally further pools the sum of the first pooling feature and the second pooling feature of all classes to obtain the first global vector.

[0112] When the multiple second unit vector sets are input to the pooling module 130, the sequential encoding module 131 is configured to respectively embed the position information of each second unit in the corresponding second modality sample into the corresponding feature vector, output multiple second sequential encoding vector sets, and the full connection module 132 is configured to respectively pool each second sequential encoding vector set, output multiple second global vectors. Specifically, the sequential encoding module 131 first generates a second position vector of each second unit based on the position information of each second unit in the corresponding second modality sample, and then takes the sum of each second position vector and the corresponding feature vector in each second modality sample as a second sequential encoding vector, thereby obtaining the multiple second sequential encoding vector sets. Specifically, the full connection module 132 first clusters each second sequential encoding vector set based on a soft assignment function to obtain multiple classes, then obtains the first pooling feature and the second pooling feature of each class corresponding to each second sequential encoding vector set respectively, and finally further pools the sum of the first pooling feature and the second pooling feature of all classes corresponding to each second sequential encoding vector set to obtain the multiple second global vectors.

[0113] The first embedding module 140 is configured to project the first global vector to a joint space to obtain a first joint vector, and the second embedding module 150 is configured to respectively project each second global vector to the same joint space to obtain multiple second joint vectors.

[0114] The acquisition module 160 is configured to acquire the cross-modality retrieval result of the first modality sample based on the similarity between the first joint vector and each second joint vector, for example, selecting a specified number of second modality samples corresponding to the second joint vectors with the highest similarity as the cross-modality retrieval result of the first modality sample.

[0115] The embodiment can improve the accuracy of the retrieval result by retaining the position information in different modality samples in the pooling process.

[0116] Based on the same inventive concept, the present application also provides a training method of a cross-modal retrieval model, please refer to Figure 10 , Figure 10 The present application is a flowchart of an embodiment of the training method of the cross-modal retrieval model, which comprises the following steps.

[0117] Step S81, input a plurality of first modal samples and a plurality of second modal samples into the retrieval model respectively, wherein the retrieval model comprises a first feature extraction network, a second feature extraction network, a pooling network, a first embedding network and a second embedding network, each first modal sample and / or each second modal sample contains label data.

[0118] Wherein, the first modal sample and the second modal sample can be text samples, image samples, audio samples, etc., and the present embodiment takes the first modal sample as a text sample and the second modal sample as an image sample as an example for illustration.

[0119] Step S82, use the first feature extraction network to extract a first unit vector set of each first modal sample, and use the second feature extraction network to extract a second unit vector set of each second modal sample, each first modal sample includes a plurality of first units, each first unit vector set is composed of feature vectors of all first units in the corresponding first modal sample, and each second modal sample includes a plurality of second units, each second unit vector set is composed of feature vectors of all second units in the corresponding second modal sample.

[0120] The present embodiment can use the first feature extraction network to extract the feature vector of each first unit to obtain the first unit vector set, and use the second feature extraction network to extract the feature vector of each second unit to obtain the second unit vector set, wherein the first unit vector set corresponds to the first modal sample one by one, and the second unit vector set corresponds to the second modal sample one by one.

[0121] For example, a text sample includes a plurality of words, an image sample includes a plurality of salient regions, the first feature extraction network includes but is not limited to a skip-gram model, and all word vectors of the text sample can be vectorized, and the second feature extraction network includes but is not limited to a Faster R-CNN model and a ResNet model, and the target of interest can be detected from the image sample, and the salient region containing the target of interest can be vectorized.

[0122] Step S83, pooling each first unit vector set using a pooling network to obtain a plurality of first global vectors, and pooling each second unit vector set using a pooling network to obtain a plurality of second global vectors, wherein each first global vector contains position information of the first unit in the corresponding first modality sample, and each second global vector contains position information of the second unit in the corresponding second modality sample.

[0123] After obtaining the first unit vector set and the second unit vector set, the embodiment uses a pooling network to pool them respectively, and the position information related to the first unit and the second unit is retained in the pooling process, thereby obtaining the first global vector and the second global vector. The first global vector corresponds to the first modality sample one-to-one, and the second global vector corresponds to the second modality sample one-to-one. For details of the pooling process, please refer to the above-mentioned embodiments related to the cross-modality retrieval method, which will not be described here.

[0124] Step S84, projecting each first global vector into a joint space using a first embedding network to obtain a plurality of first joint vectors, and projecting each second global vector into the joint space using a second embedding network to obtain a plurality of second joint vectors.

[0125] After obtaining the plurality of first global vectors and the plurality of second global vectors by pooling respectively, it is necessary to project them into the same joint space using an embedding network, and then adjust and optimize the parameters of the retrieval model. The embodiment can use the first embedding network to project each first global vector to obtain the first joint vector corresponding to the first modality sample one-to-one, and use the second embedding network to project each second global vector to obtain the second joint vector corresponding to the second modality sample one-to-one.

[0126] Step S85, adjusting the parameters of the retrieval model based on the first joint vector and the second joint vector.

[0127] The first modality sample and / or the second modality sample has label data, and the label data is strongly related to the corresponding sample. In one embodiment, please refer to Figure 11 , Figure 11 For Figure 10 The flowchart of an embodiment of step S85 in the above embodiment can be used to adjust the parameters of the retrieval model by the following steps.

[0128] Step S91, determining the first correlation degree between the first modality sample and the second modality sample using the similarity between the first joint vector and the second joint vector.

[0129] After obtaining the plurality of first joint vectors and the plurality of second joint vectors, the embodiment can obtain the similarity of two of each pair of combinations of all the first joint vectors and all the second joint vectors to determine the first correlation degree between the first modality sample and the second modality sample.

[0130] At step S92, a loss function is calculated based on the first correlation degree and a second correlation degree, wherein the second correlation degree is determined according to the label data of the first modality sample and / or the second modality sample.

[0131] After determining the first correlation degree between the first modality sample and the second modality sample, the embodiment can determine the second correlation degree according to the label data. For example, if the second modality sample has label data, the label data is vectorized, and the similarity of two of each pair of combinations of all the first joint vectors and all the label vectors is obtained to obtain the second correlation degree between the first joint vector and the label data. Then, a loss function is calculated based on the first correlation degree and the second correlation degree to adjust and optimize the retrieval model according to the loss function. The loss function includes but is not limited to a triplet loss.

[0132] At step S93, the parameters of the retrieval model are adjusted based on the loss function.

[0133] After calculating the loss function, the parameters of the retrieval model can be adjusted based on the loss function. After multiple adjustments, the retrieval model can continuously shorten the distance between the first correlation degree and the second correlation degree, so as to filter out the first modality sample and the second modality sample with the highest correlation degree, and complete the training of the retrieval model.

[0134] In the process of training the retrieval model, the embodiment retains the relevant position information in the first modality sample and the second modality sample, which can improve the accuracy of the retrieval model.

[0135] Based on the same inventive concept, the present application also provides an electronic device, please refer to Figure 12 , Figure 12 The structure of an embodiment of the electronic device of the present application is shown in the figure. The electronic device includes a memory 210 and a processor 220 coupled to each other. The memory 210 stores program instructions, and the processor 220 can execute the program instructions to implement the cross-modal retrieval method described in any of the above embodiments, or the training method of the cross-modal retrieval model described in any of the above embodiments. For details, please refer to the above embodiments, which will not be repeated here.

[0136] Based on the same inventive concept, the present application also provides a computer readable storage medium, please refer to Figure 13 , Figure 13For a structural schematic diagram of an embodiment of the computer readable storage medium of the present application, the storage medium 300 stores program instructions 310, which can be executed by a processor to implement the cross-modal retrieval method described in any of the above embodiments, or the training method of the cross-modal retrieval model described in any of the above embodiments. For details, please refer to the above embodiments, which will not be repeated here.

[0137] The above is only an embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation using the content of the specification and drawings of the present application, or direct or indirect application in other related technical fields, is also included in the patent protection scope of the present application.

Claims

1. A cross-modal retrieval method, characterized in that, include: The first modality sample, which serves as the retrieval target, and multiple second modality samples, which serve as candidate objects, are input into the retrieval model. The retrieval model includes a first feature extraction network, a second feature extraction network, a pooling network, a first embedding network, and a second embedding network. The first feature extraction network is used to extract the first unit vector set of the first modality sample, and the second feature extraction network is used to extract the second unit vector set of each second modality sample. The first modality sample includes multiple first units, and the first unit vector set is composed of the feature vectors of all the first units. Each second modality sample includes multiple second units, and each second unit vector set is composed of the feature vectors of all the second units in the corresponding second modality sample. The first unit vector set is pooled using the pooling network to obtain a first global vector, and each second unit vector set is pooled using the pooling network to obtain multiple second global vectors. The first global vector contains the position information of the first unit in the first modal sample, and each second global vector contains the corresponding position information of the second unit in the corresponding second modal sample. The first global vector is projected onto the joint space using the first embedding network to obtain a first joint vector, and each second global vector is projected onto the joint space using the second embedding network to obtain multiple second joint vectors. The cross-modal retrieval results of the first modality sample are obtained based on the similarity between the first joint vector and each of the second joint vectors; The step of pooling the first unit vector set using the pooling network to obtain the first global vector includes: The position information of each first unit in the first modality sample is embedded into the corresponding feature vector to obtain the first sequential encoding vector set; The first sequential encoded vector set is pooled using the pooling network to obtain the first global vector; The pooling network includes a first fully connected layer, a second fully connected layer, and a third fully connected layer. The step of pooling the first sequentially encoded vector set using the pooling network to obtain the first global vector includes: Clustering of the first sequential encoded vector set based on the soft allocation function yields multiple classes; The first pooling feature of each class is obtained based on the first fully connected layer, and the second pooling feature of each class is obtained based on the second fully connected layer; The sum of the first pooling features and the second pooling features of all the classes is input into the third fully connected layer to obtain the first global vector.

2. The retrieval method according to claim 1, characterized in that, The step of using the pooling network to pool each second unit vector set to obtain multiple second global vectors includes: embedding the position information of each second unit in each second modality sample into the corresponding feature vector to obtain multiple second sequential encoding vector sets; The pooling network is used to pool each of the second sequential encoded vector sets to obtain multiple second global vectors.

3. The retrieval method according to claim 2, characterized in that, The step of embedding the position information of each first unit in the first modality sample into the corresponding feature vector to obtain the first sequential encoding vector set includes: A first position vector is generated for each of the first units based on the position information of each of the first units in the first modal sample, wherein the first position vector has the same dimension as the corresponding feature vector; The feature vector of each first unit in the first unit vector set is replaced with the corresponding first sequential encoding vector to obtain the first sequential encoding vector set, wherein the first sequential encoding vector of each first unit is the sum of the first position vector of the first unit and the feature vector.

4. The retrieval method according to claim 2, characterized in that, The step involves embedding the position information of each second unit in each second modality sample into the corresponding feature vector to obtain multiple sets of second sequential encoding vectors, including: A second position vector is generated for each second unit based on the position information of each second unit in the corresponding second modality sample, and the second position vector has the same dimension as the corresponding feature vector; Replace the feature vector of each second unit in each second unit vector set with the corresponding second sequential encoding vector to obtain multiple sets of second sequential encoding vectors, wherein each second sequential encoding vector is the sum of the second position vector of the corresponding second unit and the feature vector.

5. The retrieval method according to claim 2, characterized in that, The pooling network includes a first fully connected layer, a second fully connected layer, and a third fully connected layer. The pooling network is used to pool each of the second sequential encoded vector sets to obtain multiple second global vectors, including: Clustering is performed on each of the second sequential encoding vector sets based on the soft allocation function to obtain multiple classes; Based on the first fully connected layer, the first pooling features of each class corresponding to each second sequential encoding vector set are obtained respectively, and the second pooling features of each class corresponding to each second sequential encoding vector set are obtained respectively based on the second fully connected layer; The sum of the first pooling features and the second pooling features of all classes corresponding to each second sequential encoded vector set is input into the third fully connected layer to obtain multiple second global vectors.

6. The retrieval method according to claim 1, characterized in that, The step of obtaining the cross-modal retrieval results of the first modality sample based on the similarity between the first joint vector and each of the second joint vectors includes: The second modality sample corresponding to the specified number of second joint vectors with the highest similarity is selected as the cross-modal retrieval result of the first modality sample.

7. A training method for a cross-modal retrieval model, characterized in that, include: Multiple first modal samples and multiple second modal samples are input into a retrieval model, wherein the retrieval model includes a first feature extraction network, a second feature extraction network, a pooling network, a first embedding network and a second embedding network, and each first modal sample and / or each second modal sample carries label data; The first feature extraction network is used to extract the first unit vector set of each first modality sample, and the second feature extraction network is used to extract the second unit vector set of each second modality sample. Each first modality sample includes multiple first units, and each first unit vector set is composed of the feature vectors of all the first units in the corresponding first modality sample. Each second modality sample includes multiple second units, and each second unit vector set is a set composed of the feature vectors of all the second units in the corresponding second modality sample. The pooling network is used to pool each first unit vector set to obtain multiple first global vectors, and the pooling network is used to pool each second unit vector set to obtain multiple second global vectors. Each first global vector contains the position information of the first unit in the corresponding first modality sample, and each second global vector contains the position information of the second unit in the corresponding second modality sample. The first embedding network is used to project each first global vector into the joint space to obtain multiple first joint vectors, and the second embedding network is used to project each second global vector into the joint space to obtain multiple second joint vectors. The parameters of the retrieval model are adjusted based on the first joint vector and the second joint vector; The pooling network is used to pool each of the first unit vector sets to obtain multiple first global vectors, including: The position information of each first unit in the first modality sample is embedded into the corresponding feature vector to obtain multiple first sequential encoding vector sets; The pooling network is used to pool each of the first sequential encoded vector sets to obtain multiple first global vectors; The pooling network includes a first fully connected layer, a second fully connected layer, and a third fully connected layer. The pooling network is used to pool each of the first sequential encoded vector sets to obtain multiple first global vectors, including: Clustering is performed on each of the first sequential encoded vector sets based on the soft allocation function to obtain multiple classes; The first pooling feature of each class is obtained based on the first fully connected layer, and the second pooling feature of each class is obtained based on the second fully connected layer; The sum of the first pooling features and the second pooling features of all the classes is input into the third fully connected layer to obtain multiple first global vectors.

8. The training method according to claim 7, characterized in that, The step of adjusting the parameters of the retrieval model based on the first joint vector and the second joint vector includes: The first correlation degree between the first modality sample and the second modality sample is determined by using the similarity between the first joint vector and the second joint vector; A loss function is calculated based on the first correlation degree and the second correlation degree, wherein the second correlation degree is determined based on the label data of the first modality sample and / or the second modality sample; The parameters of the retrieval model are adjusted based on the loss function.

9. An electronic device, characterized in that, It includes a memory and a processor coupled to each other, the memory storing program instructions, and the processor being able to execute the program instructions to implement the cross-modal retrieval method of any one of claims 1-6, or the training method of the cross-modal retrieval model of any one of claims 7-8.

10. A computer-readable storage medium, characterized in that, The storage medium stores program instructions that can be executed by a processor to implement the cross-modal retrieval method of any one of claims 1-6, or the training method of the cross-modal retrieval model of any one of claims 7-8.

11. An electronic device, characterized in that, The system includes a memory and a processor that are coupled to each other. The memory stores program instructions, and the processor is capable of executing the program instructions to: input a first modality sample as a retrieval target and a plurality of second modality samples as candidate objects into a retrieval model. The retrieval model includes a first feature extraction network, a second feature extraction network, a pooling network, a first embedding network, and a second embedding network. The first feature extraction network is used to extract the first unit vector set of the first modality sample, and the second feature extraction network is used to extract the second unit vector set of each second modality sample. The first modality sample includes multiple first units, and the first unit vector set is composed of the feature vectors of all the first units. Each second modality sample includes multiple second units, and each second unit vector set is composed of the feature vectors of all the second units in the corresponding second modality sample. The first unit vector set is pooled using the pooling network to obtain a first global vector, and each second unit vector set is pooled using the pooling network to obtain multiple second global vectors. The first global vector contains the position information of the first unit in the first modal sample, and each second global vector contains the corresponding position information of the second unit in the corresponding second modal sample. The first global vector is projected onto the joint space using the first embedding network to obtain a first joint vector, and each second global vector is projected onto the joint space using the second embedding network to obtain multiple second joint vectors. The cross-modal retrieval results of the first modality sample are obtained based on the similarity between the first joint vector and each of the second joint vectors; The step of pooling the first unit vector set using the pooling network to obtain the first global vector includes: The position information of each first unit in the first modality sample is embedded into the corresponding feature vector to obtain the first sequential encoding vector set; The first sequential encoded vector set is pooled using the pooling network to obtain the first global vector; The pooling network includes a first fully connected layer, a second fully connected layer, and a third fully connected layer. The step of pooling the first sequentially encoded vector set using the pooling network to obtain the first global vector includes: Clustering of the first sequential encoded vector set based on the soft allocation function yields multiple classes; The first pooling feature of each class is obtained based on the first fully connected layer, and the second pooling feature of each class is obtained based on the second fully connected layer; The sum of the first pooling features and the second pooling features of all the classes is input into the third fully connected layer to obtain the first global vector.

12. The electronic device according to claim 11, characterized in that, The pooling network is used to pool each of the second unit vector sets to obtain multiple second global vectors, including: The position information of each second unit in each second modality sample is embedded into the corresponding feature vector to obtain multiple sets of second sequential encoding vectors; The pooling network is used to pool each of the second sequential encoded vector sets to obtain multiple second global vectors.

13. The electronic device according to claim 12, characterized in that, The step of embedding the position information of each first unit in the first modality sample into the corresponding feature vector to obtain the first sequential encoding vector set includes: A first position vector is generated for each of the first units based on the position information of each of the first units in the first modal sample, wherein the first position vector has the same dimension as the corresponding feature vector; The feature vector of each first unit in the first unit vector set is replaced with the corresponding first sequential encoding vector to obtain the first sequential encoding vector set, wherein the first sequential encoding vector of each first unit is the sum of the first position vector of the first unit and the feature vector.

14. The electronic device according to claim 12, characterized in that, The step involves embedding the position information of each second unit in each second modality sample into the corresponding feature vector to obtain multiple sets of second sequential encoding vectors, including: A second position vector is generated for each second unit based on the position information of each second unit in the corresponding second modality sample, and the second position vector has the same dimension as the corresponding feature vector; Replace the feature vector of each second unit in each second unit vector set with the corresponding second sequential encoding vector to obtain multiple sets of second sequential encoding vectors, wherein each second sequential encoding vector is the sum of the second position vector of the corresponding second unit and the feature vector.

15. The electronic device according to claim 12, characterized in that, The pooling network includes a first fully connected layer, a second fully connected layer, and a third fully connected layer. The pooling network is used to pool each of the second sequential encoded vector sets to obtain multiple second global vectors, including: Clustering is performed on each of the second sequential encoding vector sets based on the soft allocation function to obtain multiple classes; Based on the first fully connected layer, the first pooling features of each class corresponding to each second sequential encoding vector set are obtained respectively, and the second pooling features of each class corresponding to each second sequential encoding vector set are obtained respectively based on the second fully connected layer; The sum of the first pooling features and the second pooling features of all classes corresponding to each second sequential encoded vector set is input into the third fully connected layer to obtain multiple second global vectors.

16. The electronic device according to claim 11, characterized in that, The step of obtaining the cross-modal retrieval results of the first modality sample based on the similarity between the first joint vector and each of the second joint vectors includes: The second modality sample corresponding to the specified number of second joint vectors with the highest similarity is selected as the cross-modal retrieval result of the first modality sample.

17. An electronic device, characterized in that, It includes a memory and a processor coupled to each other, wherein the memory stores program instructions and the processor is capable of executing the program instructions to implement: Multiple first modal samples and multiple second modal samples are input into a retrieval model, wherein the retrieval model includes a first feature extraction network, a second feature extraction network, a pooling network, a first embedding network and a second embedding network, and each first modal sample and / or each second modal sample carries label data; The first feature extraction network is used to extract the first unit vector set of each first modality sample, and the second feature extraction network is used to extract the second unit vector set of each second modality sample. Each first modality sample includes multiple first units, and each first unit vector set is composed of the feature vectors of all the first units in the corresponding first modality sample. Each second modality sample includes multiple second units, and each second unit vector set is a set composed of the feature vectors of all the second units in the corresponding second modality sample. The pooling network is used to pool each first unit vector set to obtain multiple first global vectors, and the pooling network is used to pool each second unit vector set to obtain multiple second global vectors. Each first global vector contains the position information of the first unit in the corresponding first modality sample, and each second global vector contains the position information of the second unit in the corresponding second modality sample. The first embedding network is used to project each first global vector into the joint space to obtain multiple first joint vectors, and the second embedding network is used to project each second global vector into the joint space to obtain multiple second joint vectors. The parameters of the retrieval model are adjusted based on the first joint vector and the second joint vector; The pooling network is used to pool the first unit vector set to obtain multiple first global vectors, including: The position information of each first unit in the first modality sample is embedded into the corresponding feature vector to obtain multiple first sequential encoding vector sets; The pooling network is used to pool each of the first sequential encoded vector sets to obtain multiple first global vectors; The pooling network includes a first fully connected layer, a second fully connected layer, and a third fully connected layer. The pooling network is used to pool each of the first sequential encoded vector sets to obtain multiple first global vectors, including: Clustering is performed on each of the first sequential encoded vector sets based on the soft allocation function to obtain multiple classes; The first pooling feature of each class is obtained based on the first fully connected layer, and the second pooling feature of each class is obtained based on the second fully connected layer; The sum of the first pooling features and the second pooling features of all the classes is input into the third fully connected layer to obtain multiple first global vectors.

18. The electronic device according to claim 17, characterized in that, The step of adjusting the parameters of the retrieval model based on the first joint vector and the second joint vector includes: determining the first correlation degree between the first modality sample and the second modality sample using the similarity between the first joint vector and the second joint vector; A loss function is calculated based on the first correlation degree and the second correlation degree, wherein the second correlation degree is determined based on the label data of the first modality sample and / or the second modality sample; The parameters of the retrieval model are adjusted based on the loss function.

Images