A cross-modal retrieval method and system for data processing

Abstract

The embodiment of the present invention discloses a cross-modal retrieval method for data processing, which relates to the field of graphic and text data processing, and can reduce information loss while having strong cross-modal similarity. The present invention includes: inputting the sample data to be processed into a modality-specific preprocessing module and performing feature extraction to obtain feature vector information. The obtained feature vector information is mapped to a common subspace through a modality-specific subnetwork. The common representation in the common subspace is mapped to a semantic subspace through a first network. The common representation in the common subspace is constructed through a second network to construct a modality discriminator, and the constructed modality discriminator is used to distinguish the original modality of each common representation. The present invention is suitable for mutual retrieval between graphic and text fields.

Description

Technical Field

[0001] The present invention relates to the field of graphic and text data processing, and in particular to a cross-modal retrieval method and system for data processing. Background Art

[0002] Cross-modal retrieval refers to retrieving related data of another type by using one type of data as a query. Its flexibility in retrieving across different modalities (such as images and text) has been widely concerned by academia and industry. Calculating the correlation between multimodal data is the core goal of cross-modal retrieval. However, the potential heterogeneity between modalities leads to incomparability, so the key to achieving cross-modal retrieval lies in how to bridge the heterogeneous gap between different modalities.

[0003] A common method to eliminate cross-modal differences is representation learning, which transforms data from different modalities into a common subspace by learning a modality-specific transformation function, in which similarity measurements can be directly performed. However, existing schemes usually only focus on part of the information in the dataset during the learning transformation process, and there are different degrees of information loss in the design of the objective function, which limits the model performance.

[0004] Therefore, how to have strong cross-modal similarity while reducing information loss has become a topic that needs to be studied. Summary of the invention

[0005] Embodiments of the present invention provide a cross-modal retrieval method and system for data processing, which can reduce information loss while having strong cross-modal similarity.

[0006] To achieve the above object, the embodiments of the present invention adopt the following technical solutions:

[0007] In one aspect, a cross-modal retrieval method for data processing is provided, comprising:

[0008] The unprocessed image and text are input as input sample pairs into the respective modality-specific preprocessing modules for feature extraction, and the original high-dimensional image feature vector and text feature vector are obtained respectively.

[0009] The original feature vectors of the two modalities are nonlinearly mapped to a common subspace through their respective modality-specific subnetworks. Based on the common subspace, a cross-modal information aggregation constraint is proposed, which aggregates the global information and fine-grained information in the dataset while considering both absolute distance and relative distance.

[0010] The common representation in the common subspace is further mapped to a semantic subspace through the first network, and a semantic constraint is proposed based on the potential association between the vector representation in the semantic subspace and the sample label.

[0011] The common representation in the common subspace is further passed through a second network to construct a modal discriminator. Based on the modal discriminator, a modal constraint is proposed to distinguish the original modality of each common representation. Its optimization goal is opposite to that of the cross-modal information aggregation constraint. The two rely on modal feature information to confront each other and introduce adversarial learning into the model.

[0012] In another aspect, a cross-modal retrieval system for data processing is provided, comprising:

[0013] A preprocessing module, used for inputting the sample data to be processed into the modality-specific preprocessing module and performing feature extraction to obtain feature vector information, wherein the sample data to be processed includes a sample pair consisting of image data and text data, and the obtained feature vector information includes: an image feature vector and a text feature vector;

[0014] A processing module, used for mapping the obtained feature vector information to a common subspace through a modality-specific subnetwork, wherein a cross-modal information aggregation constraint is obtained based on the common subspace correspondence; mapping the common representation in the common subspace to a semantic subspace through a first network, wherein a potential association is established between the vector representation in the semantic subspace and the sample label, and the potential association corresponds to a semantic constraint; constructing a modality discriminator for the common representation in the common subspace through a second network, and using the constructed modality discriminator to distinguish the original modality of each common representation; and storing the result of distinguishing the original modality of each common representation in a database module;

[0015] A database module, used for storing the modal results output by the processing module;

[0016] The query feedback module is used to receive the query items sent by the terminal device, convert the query items into a public representation, and then query the modal results stored in the database to obtain the public representation that is most similar to the converted public representation and comes from another modality, and feedback the query results to the terminal device. The present invention uses dual constraints of two subspaces and adversarial learning to minimize information loss in the cross-modal process, and generates a public representation with stronger cross-modal similarity and semantic distinction, thereby achieving strong cross-modal similarity while reducing information loss. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A schematic diagram of a possible implementation of a cross-modal retrieval model provided in an embodiment of the present invention.

[0019] Figure 2 A schematic diagram of a possible implementation of the first model provided in an embodiment of the present invention.

[0020] Figure 3 A schematic diagram of a possible implementation of the second model provided in an embodiment of the present invention.

[0021] Figure 4 A schematic diagram of a method flow chart provided by an embodiment of the present invention.

[0022] Figure 5 A schematic diagram of a system architecture provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The embodiments of the present invention will be described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention. It can be understood by those skilled in the art that, unless specifically stated, the singular forms "one", "one", "said" and "the" used herein may also include plural forms. It should be further understood that the term "including" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or their groups. It should be understood that when we say that an element is "connected" or "coupled" to another element, it can be directly connected or coupled to other elements, or there may also be intermediate elements. In addition, the "connection" or "coupling" used here may include wireless connection or coupling. The term "and / or" used herein includes any unit and all combinations of one or more associated listed items. It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as generally understood by those of ordinary skill in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0024] The embodiment of the present invention provides a cross-modal retrieval method for data processing, specifically, an improved solution belonging to the cross-modal retrieval technology, and the main design idea is: Figure 1 As shown in the figure, the dual constraints of the two subspaces and adversarial learning are used to minimize the information loss in the cross-modal process and generate a common representation with stronger cross-modal similarity and semantic distinction. The main method flow is as follows Figure 1 , 4 As shown, including:

[0025] S1. Input the sample data to be processed into a modality-specific preprocessing module and perform feature extraction to obtain feature vector information.

[0026] Among them, the modality-specific preprocessing module refers to the use of specific different preprocessing models for specific modalities. For image modality data, we use the VGG network to perform preprocessing, and for text modality data, we use the BoW network to perform preprocessing. For details, see Figure 2 Among them, the VGG network is a neural network that is used for deep learning to extract feature information from images, and the effect is very good. BoW is also a common model for processing text.

[0027] The preprocessing module mentioned in this embodiment can be implemented as a code program in actual application, as shown in the attached Figure 2 As shown in the figure, we use different models for different modalities to extract features from original images and texts. The VGG network is a neural network that specializes in processing images in deep learning. It has very good results and is widely used in the field of artificial intelligence. We regard it as a preprocessing module for image modality. BoW is also a common and widely used model for processing text. We regard it as a preprocessing module for text modality.

[0028] The sample data to be processed includes a sample pair consisting of image data and text data, and the obtained feature vector information includes: an image feature vector and a text feature vector. For example: unprocessed images and texts can be used as input sample pairs, and input into their respective modality-specific preprocessing modules for feature extraction, to obtain the original high-dimensional image feature vectors and text feature vectors, respectively. It should be noted that there is no specific standard for high and low. The high dimension referred to in this embodiment is generally several thousand dimensions, and the low dimension is generally several hundred dimensions. Both high and low are relative. The outputs of BOW and VGG are three to four thousand dimensions, while the common subspace dimensions are only a few hundred.

[0029] S2. Map the obtained feature vector information to the common subspace through the modality-specific sub-network.

[0030] Among them, a cross-modal information aggregation constraint is obtained based on the common subspace correspondence. For example, the original feature vectors of the two modalities can be nonlinearly mapped to a common subspace through the subnetworks specific to each modality, and a cross-modal information aggregation constraint is proposed based on the common subspace, which aggregates the global information and fine-grained information in the data set while considering both absolute distance and relative distance.

[0031] Specifically, the cross-modal information aggregation constraint refers to: a loss function (some scholars also call it retrieval loss) is proposed based on the common representation of the common subspace. The formula of the loss function is the following L r This part is called cross-modal information aggregation because the retrieval loss in this part aggregates the global information and fine-grained information in the dataset. It sounds more intuitive than the retrieval loss. The retrieval loss is a short name or a specific implementation function of the cross-modal information aggregation constraint. Semantic constraint refers to: Similar to the above, a semantic loss function is proposed based on the vector representation in the semantic subspace. The function formula is the following L s The modal constraint refers to: the loss function of the modal discriminator is called the modal loss, and the formula is the following L d In actual design, each constraint corresponds to a specific loss function.

[0032] S3. Map the common representation in the common subspace to a semantic subspace through a first network.

[0033] Among them, a potential association is established between the vector representation in the semantic subspace and the sample label, and the potential association corresponds to a semantic constraint. For example: the common representation in the common subspace can be further mapped to a semantic subspace through the first network, and a semantic constraint is proposed based on the potential association between the vector representation in the semantic subspace and the sample label. Specifically, the common subspace is obtained from S2. For example: in s2, the image feature vector (such as 4096 dimensions) is input into the image subnetwork, and a 200-dimensional vector is output after network mapping. Then the 200-dimensional vector can be regarded as a point in a 200-dimensional space, and the vector is the coordinate of the point. Therefore, after step S2, all feature vectors are mapped to a point in the 200-dimensional space. Because both the image and the text are in this space, the space is called a common subspace, and each point in the space (that is, each 200-dimensional vector) is called a common representation. The first network and the second network are shown in the attached figure. Figure 1 and 3The first network consists of one fully connected layer. Just like the common subspace above, its output vector can also be regarded as a point in the space (this space is called the semantic subspace). The second network consists of three fully connected layers. The modality discriminator is an alias I gave to the function of the second network. Its output is no longer a vector, but a value (scalar), which can be used to discriminate the modality.

[0034] S4. Using the common representations in the common subspace, construct a modality discriminator through a second network, and use the constructed modality discriminator to distinguish the original modality of each common representation.

[0035] The working mode of this embodiment is similar to that of a commonly used search engine (such as an image search engine). When a query item is input, the model will first convert the query item into a public representation, and then the model will search for a public representation from another modality that is most similar to the public representation in, for example, a database, and then return the corresponding query result to the user. For example, when the user enters the word cat, the model returns some cat pictures to her. The user enters a picture, and the model returns some reports or descriptions about the picture to her. Among them, the public representation in the public subspace can be further constructed into a modal discriminator through a second network, and a modal constraint is proposed based on the modal discriminator to distinguish the original modality of each public representation, which is opposite to the optimization goal of the cross-modal information aggregation constraint. The two rely on the modal feature information to confront each other to introduce adversarial learning into the model. The present invention adopts the dual constraints of the two subspaces and adversarial learning to minimize the information loss in the cross-modal process, and generates a public representation with stronger cross-modal similarity and semantic distinction.

[0036] In this embodiment, step S1 includes: extracting features from image data through the VGG-19 network, wherein the 4096-dimensional vector output from the fc7 layer is obtained as the input of the image sub-network in the second model. The text data is processed through the bag-of-words model (BoW), and a high-dimensional text feature vector is generated as the input of the text sub-network in the second model. For example: Figure 2As shown, the unprocessed image and text are input as input sample pairs into the respective modality-specific preprocessing modules for feature extraction, and the original high-dimensional image feature vector and text feature vector are obtained as the input of the subsequent sub-network. Although it is the first half of the entire cross-modal retrieval model, it does not participate in the training of the entire model. It is essentially a data preprocessing process. Specifically, for unprocessed image samples, this embodiment uses a pre-trained VGG-19 network to extract features, and obtains the 4096-dimensional vector output in its fc7 layer as the input of the image sub-network in the second model. The unprocessed text samples are generated through the well-known bag of words model (BoW) to generate the original high-dimensional text feature vector as the input of the text sub-network in the second model.

[0037] Specifically, in the process of constructing the common subspace, the original feature vectors of the two modalities can be nonlinearly mapped to a common subspace through the subnetworks specific to each modality. For example: the image subnetwork and the text subnetwork are respectively composed of three layers of fully connected neural networks. The fully connected neural network has rich parameters and the subnetwork has sufficient capacity to realize this complex transformation. In order to better convert from different modalities to the same subspace, this embodiment adopts weight sharing in the last layer of the two subnetworks. Based on the common subspace, this embodiment proposes a cross-modal information aggregation constraint. For simplicity, this embodiment calls the constraint a retrieval loss, which aggregates the global information and fine-grained information in the data set while considering both absolute distance and relative distance, greatly reducing the information loss in the cross-modal process, and maximizing the cross-modal similarity of the common representation.

[0038] In this embodiment, step S2 includes:

[0039] The obtained image feature vector and text feature vector are nonlinearly mapped to the common subspace through their respective modality-specific subnetworks, wherein the image subnetwork and the text subnetwork are respectively composed of three-layer fully connected neural networks, and the retrieval loss model used in the mapping process consists of three sub-items.

[0040] The first sub-term is constructed with triple center loss, where the triple is where t q is a text query term, is the positive class center and is consistent with the text query item t q The tag categories are the same. is the negative class center and is consistent with the text query item t q The tag categories are not the same.

[0041] The triplet center loss is: Among them, N 1 Represents the total number of triplets, m1 represents an adjustable threshold, i 1 、i 2 and i 3 The 1, 2, and 3 in the subscripts are just to distinguish different class centers. i indicates that the item belongs to the image mode, and t indicates that the item belongs to the text mode. i = image, t = text.

[0042] For example: The specific design of retrieval loss includes: the retrieval loss consists of three sub-items, which appear in the form of triples, quadruples and pairs. It aggregates global information and fine-grained information in the data set. Among them: Step 1, construct the first sub-item, namely the triple center loss. Next, take the text query image as an example: a triple is defined as where t q is a text query term, and are two different image category centers. This embodiment is called The positive class center is related to the text query term t q The tag categories are the same, in this embodiment, is the negative class center, which is related to the text query item t q The label categories are not the same. Then, the triple center loss is defined as follows:

[0043]

[0044] Among them, N 1 Represents the total number of triplets, m 1 is an adjustable threshold, [A] + =max(0,A) is the hinge function, d(a,b) = ||ab|| 2 Represents the Euclidean distance. The above formula assumes that the distance from the query item to the corresponding positive class center is less than the distance from the query item to any negative class center. This sub-item takes into account the absolute distance under the same anchor point and uses the relationship between the sample and the class center to introduce global information to the model.

[0045] The quadruple center loss constructs the second sub-item, where the quadruple is is with For different negative class centers, the quadruple center loss is: Among them, N 2 Indicates the total number of quadruples, m 2 is another adjustable threshold. For example: Step 2, construct the second sub-item, the quadruple center loss. A quadruple is formally defined as and are any two different negative class centers. Then, the quadruple center loss is defined as follows:

[0046]

[0047] Among them, N 2 Indicates the total number of quadruples, m 2 is another adjustable threshold. The above formula ensures that the distance between the query item and the positive class center is less than the distance between any two different negative class centers. This sub-item considers the relative distance under different anchor points and introduces another part of the global information in the dataset to the model as a supplement to the triple center loss.

[0048] Using triple center loss and quad center loss, we establish global constraints: Among them, σ I and σ T are the weight parameters of the image sub-network and the text sub-network respectively. For example: Step 3, by combining the above two formulas, this embodiment obtains a constraint at the global level, and the formula is defined as follows:

[0049]

[0050] Among them, σ I and σ T are the weight parameters of the image subnetwork and the text subnetwork respectively. The above formula enables the model to construct a common subspace using global information, which greatly reduces the difficulty of model updating due to the huge differences between sample pairs. Similarly, this embodiment can obtain the image query text case

[0051]

[0052] Build the third sub-item: in, E is an indicator matrix, i j represents the jth image sample and t k represents the kth text sample, j and k represent positive integers and respectively, i represents that the item belongs to the image mode, and t represents that the item belongs to the text mode. i = image, t = text. J and k represent the sample serial numbers respectively. So i j represents the jth image sample, t k The kth text sample, E jk Used to represent i j and t k Are the categories the same? If so, then E jk =1, otherwise E jk= 0. For example: Step 4, construct the third sub-item, which is a constraint at the local level based on sample pairs, and the formula is defined as:

[0053]

[0054] in, E is an indicator matrix, E jk Indicates i j and t k If the two samples belong to the same category, then E jk =1, otherwise E jk = 0. Formula 4 makes the cosine value between samples of the same category as large as possible, and the cosine value between samples of different categories as small as possible.

[0055] According to the constraints on the global level and the third sub-item, the complete retrieval loss is constructed: Where γ is a hyperparameter. For example: Step 5, by combining the formulas of Step 3 and Step 4, this embodiment obtains the complete retrieval loss, and the formula is defined as follows:

[0056]

[0057] where γ is a hyperparameter. This formula effectively reduces the information loss by aggregating constraints at different levels and retains the cross-modal similarity of the common representation.

[0058] In this embodiment, in step S3, the first network is composed of a layer of fully connected neural network, in which the semantic constraints are: Among them, σ s represents a network parameter of the first network, represents the vector representation in the semantic subspace, do represents the number of categories of samples in the training dataset, and R do represents the vector space of do dimensions, Represents the label vector of the corresponding sample. For example, the common representation in the common subspace can be further mapped to a semantic subspace through the first network, and a semantic constraint is proposed based on the semantic subspace, which uses the potential semantic association between the sample label and the vector representation of the semantic subspace to optimize the distribution of the common subspace. Figure 3 , the first network consists of a layer of fully connected neural network. The semantic constraint formula is expressed as follows:

[0059]

[0060] where σ s is the network parameter of this part, is the vector representation in the semantic subspace, is the label vector of the corresponding sample. This formula not only improves the semantic discriminability of the public representation, but also plays a certain regularization role in the generation process of the public representation.

[0061] In this embodiment, in step S4, the modal loss of the modal discriminator is established:

[0062] f bce (x,c)=c(x)log(p(f T (t x ; σ T )))+(1-c(x))log(p(f I (i x ; σ I )))

[0063]

[0064] Among them, f bce represents the binary cross entropy loss function for modality classification, c(·) represents a modality indicator, when the input x represents text, c(x) = 1, otherwise c(x) = 0, p(·) represents the probability of each modality generated by the input, σ d Represents the parameters of the discriminator. For example, the common representation in the common subspace can be further used to construct a modal discriminator through a second network. This embodiment proposes a modal constraint based on the modal discriminator to distinguish the original modality of each common representation, and its goal is to distinguish the original modalities of the common representation as much as possible. However, the cross-modal aggregation constraint aims to generate a common representation with cross-modal similarity, which is contrary to the purpose of the modal discriminator. Therefore, the two compete in a maximum-minimum game. For simplicity, the loss of the discriminator is called the modal loss, and the modal loss formula is defined as follows:

[0065] f bce (x,c)=c(x)log(p(f T (t x ; σ T )))+(1-c(x))log(p(f I (i x ; σ I )))

[0066]

[0067] where f bce is the binary cross entropy loss function for modality classification. c(·) is a modality indicator, when the input x is text, c(x) = 1, otherwise c(x) = 0. p(·) is the probability of each modality generated by the input, σ d are the parameters of the discriminator. By maximizing This further improves the performance and robustness of the model.

[0068] Furthermore, in this embodiment, the modality discriminator may be optimized by the Adam algorithm, wherein, during the optimization process, a maximum-minimum game is performed by two parallel subprocesses, including:

[0069]

[0070]

[0071] Specifically, the Adam algorithm can be used, which approximates the true gradient of the mini-batch to update the model. The process of learning the best representation is to jointly minimize the retrieval loss, semantic loss, and modality loss. Since the optimization goals of the retrieval loss and the modality discriminator are opposite, the process is run as a maximum-minimum game of two parallel sub-processes with the following formula:

[0072]

[0073]

[0074] The training of this model is actually a process of continuously optimizing k generation processes and one discrimination process until the results of the entire model converge. As with all adversarial learning methods, the parameters of the "generator" are fixed during the training phase of the "discriminator", and vice versa. Figure 3 It can be seen that the method of the present invention can greatly reduce the information loss in the cross-modal process, generate a public representation with stronger cross-modal similarity and semantic discriminability, and effectively improve the accuracy of cross-modal retrieval.

[0075] The advantages of this embodiment are: information loss is greatly reduced through cross-modal information aggregation constraints, and the generated common representation has stronger cross-modal similarity. Semantic constraints use the semantic information in sample labels to improve the semantic discriminability of the common representation. Modal constraints use modality-fixed information to further reduce information loss and enhance model robustness.

[0076] It should be noted that this embodiment is not a simple calculation method, but can be applied to a retrieval system and assist in improving a search engine. Figure 5 The system shown comprises:

[0077] The preprocessing module is used to input the sample data to be processed into a modality-specific preprocessing module and perform feature extraction to obtain feature vector information, wherein the sample data to be processed includes a sample pair consisting of image data and text data, and the obtained feature vector information includes: image feature vector and text feature vector.

[0078] A processing module is used to map the obtained feature vector information to a common subspace through a modality-specific subnetwork, wherein a cross-modal information aggregation constraint is obtained based on the common subspace correspondence; the common representation in the common subspace is mapped to a semantic subspace through a first network, wherein a potential association is established between the vector representation in the semantic subspace and the sample label, and the potential association corresponds to a semantic constraint; the common representation in the common subspace is constructed into a modality discriminator through a second network, and the constructed modality discriminator is used to distinguish the original modality of each common representation; and the result of distinguishing the original modality of each common representation is stored in a database module.

[0079] The database module is used to store the modal results output by the processing module.

[0080] The query feedback module is used to receive query items sent by the terminal device, convert the query items into public representations, then query the modal results stored in the database, obtain the public representation that is most similar to the converted public representation and comes from another modality, and feed back the query results to the terminal device.

[0081] Specifically, this embodiment is suitable for mutual retrieval between graphic and text fields, that is, the query item will be converted into a public representation through the trained model, and the model will further return the query result from another modality by measuring the similarity between the query item and other public representations. For example: the working method is similar to the currently commonly used search engine. When a query item is entered, the model will first convert the query item into a public representation, and then the model will search for the public representation from another modality that is most similar to the public representation in, for example, a database, and then return the corresponding query result.

[0082] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The above is only a specific implementation method of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or replacements that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A cross-modal retrieval method for data processing, characterized in that: include: S1. Inputting the sample data to be processed into a modality-specific preprocessing module and performing feature extraction to obtain feature vector information, wherein the sample data to be processed includes a sample pair consisting of image data and text data, and the obtained feature vector information includes: an image feature vector and a text feature vector; S2, mapping the obtained feature vector information to a common subspace through a modality-specific subnetwork, wherein cross-modal information aggregation constraints are obtained based on the common subspace correspondence; S3, mapping the common representation in the common subspace to the semantic subspace through the first network, wherein a potential association is established between the vector representation in the semantic subspace and the sample label, and the potential association corresponds to a semantic constraint; S4, constructing a modality discriminator for the common representations in the common subspace through a second network, and using the constructed modality discriminator to distinguish the original modality of each common representation; Step S2 includes: The obtained image feature vector and text feature vector are nonlinearly mapped to the common subspace through their respective modality-specific subnetworks, wherein the image subnetwork and the text subnetwork are respectively composed of three layers of fully connected neural networks, and the retrieval loss model used in the mapping process is composed of three sub-items; Also includes: The first sub-term is constructed with triple center loss, where the triple is where t q is a text query term, is the positive class center and is consistent with the text query item t q The tag categories are the same. is the negative class center and is consistent with the text query item t q The label categories are not the same; The triplet center loss is: Among them, N1 represents the total number of triplets, m1 represents an adjustable threshold, and i1, i2, and i3 represent different class centers respectively; The quadruple center loss constructs the second sub-item, where the quadruple is is with For different negative class centers, the quadruple center loss is: Among them, N2 represents the total number of quadruples, and m2 is another adjustable threshold; Using triple center loss and quad center loss, we establish global constraints: Among them, σ I and σ T are the weight parameters of the image sub-network and the text sub-network respectively; Also includes: Build the third sub-item: in, E is an indicator matrix, i j represents the jth image sample and t k They represent the kth text sample, j and k represent positive integers, E jk Used to represent i j and t k Are the categories the same? If so, then E jk =1, otherwise E jk =0; According to the constraints on the global level and the third sub-item, the complete retrieval loss is constructed: Among them, γ is a hyperparameter.

2. The method according to claim 1, characterized in that After step S4, the method further includes: After receiving the query item sent by the terminal device, converting the query item into a public representation; querying the modality results stored in the database and obtaining the public representation from another modality that is most similar to the converted public representation; Feedback the query result to the terminal device.

3. The method according to claim 1, characterized in that Step S1 includes: Perform feature extraction on the image data through the VGG-19 network, wherein the 4096-dimensional vector output in the fc7 layer is obtained as the input of the image sub-network in the second model; The text data is processed through the bag-of-words (BoW) model, and a high-dimensional text feature vector is generated as the input of the text sub-network in the second model.

4. The method according to claim 1, characterized in that In step S3, the first network is composed of a layer of fully connected neural network, in which the semantic constraints are: Among them, σ s represents a network parameter of the first network, represents the vector representation in the semantic subspace, do represents the number of categories of samples in the training dataset, and R do Represents the vector space of do dimensions, indicating Represents the label vector corresponding to the sample.

5. The method according to claim 1, characterized in that In step S4, the modal loss of the modal discriminator is established: f bce (x,c)=c(x)log(p(f T (t x ;σ T )))+(1-c(x))log(p(f I (i x ;σ I ))) Among them, f bce represents the binary cross entropy loss function for modality classification, c(·) represents a modality indicator, when the input x represents text, c(x) = 1, otherwise c(x) = 0, p(·) represents the probability of each modality generated by the input, σ d Represents the parameters of the discriminator.

6. The method according to claim 5, characterized in that Also includes: The modality discriminator is optimized by the Adam algorithm, wherein, during the optimization process, a maximum-minimum game is performed by two parallel subprocesses, including:

7. A cross-modal retrieval system for data processing, characterized in that: include: A preprocessing module, used for inputting the sample data to be processed into the modality-specific preprocessing module and performing feature extraction to obtain feature vector information, wherein the sample data to be processed includes a sample pair consisting of image data and text data, and the obtained feature vector information includes: an image feature vector and a text feature vector; A processing module, used for mapping the obtained feature vector information to a common subspace through a modality-specific subnetwork, wherein a cross-modal information aggregation constraint is obtained based on the common subspace correspondence; mapping the common representation in the common subspace to a semantic subspace through a first network, wherein a potential association is established between the vector representation in the semantic subspace and the sample label, and the potential association corresponds to a semantic constraint; constructing a modality discriminator for the common representation in the common subspace through a second network, and using the constructed modality discriminator to distinguish the original modality of each common representation; and storing the result of distinguishing the original modality of each common representation in a database module; A database module, used for storing the modal results output by the processing module; A query feedback module, configured to receive a query item sent by a terminal device, convert the query item into a public representation, then query the modality results stored in the database, obtain a public representation from another modality that is most similar to the converted public representation, and feed back the query result to the terminal device; Step S2 includes: nonlinearly mapping the obtained image feature vector and text feature vector to the common subspace through their respective modality-specific subnetworks, wherein the image subnetwork and the text subnetwork are respectively composed of three layers of fully connected neural networks, and the retrieval loss model used in the mapping process is composed of three sub-items; It also includes: constructing the first sub-item with triple center loss, where the triple is where t q is a text query term, is the positive class center and is consistent with the text query item t q The tag categories are the same. is the negative class center and is consistent with the text query item t q The label categories are different; the triple center loss is: Among them, N1 represents the total number of triplets, m1 represents an adjustable threshold, i1, i2 and i3 represent different class centers respectively; the quadruple center loss constructs the second sub-item, where the quadruple is is with For different negative class centers, the quadruple center loss is: Among them, N2 represents the total number of quadruplets, and m2 is another adjustable threshold. The constraints at the global level are established by using the triple center loss and the quad center loss: Among them, σ I and σ T are the weight parameters of the image sub-network and the text sub-network respectively; Also includes: Build the third subitem: in, E is an indicator matrix, i j represents the jth image sample and t k They represent the kth text sample, j and k represent positive integers, E jk Used to represent i j and t k Are the categories the same? If so, then E jk =1, otherwise E jk =0; according to the constraints on the global level, the third sub-item is constructed to construct a complete retrieval loss: Among them, γ is a hyperparameter.

Images