Weakly supervised cross-modal object retrieval method based on multi-view relation consistency

Abstract

The present invention relates to the field of computer vision and pattern recognition technology, and in particular to a weakly supervised cross-modal target retrieval method based on multi-view relationship consistency. The present invention constructs a multi-view hypergraph and performs relationship consistency reasoning based on a label propagation algorithm according to the multi-view hypergraph, which can effectively fuse sample relationships from multiple views, make the information of multiple views complementary, and mine the consistency in multi-view sample relationships. It can realize multi-view-based training of weakly supervised cross-modal models and improve the robustness of the model, so that better cross-modal target retrieval performance can be obtained for diverse, heterogeneous and complex multi-modal data according to the weakly supervised cross-modal model.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and pattern recognition, and in particular to a weakly supervised cross-modal target retrieval method based on multi-view relationship consistency. Background Art

[0002] Cross-modal object retrieval aims to associate target images such as people and cars with corresponding query text descriptions, and is widely used in smart cities and smart security. However, since multimodal data is usually diverse, heterogeneous, and complex, it is not easy to obtain a large amount of annotated data. Therefore, a weakly supervised cross-modal object retrieval task was proposed, which only includes text-image pairs without identity annotations. This task faces difficulties such as complex intra-modal changes and cross-modal matching ambiguity.

[0003] The existing text-based unsupervised pedestrian target retrieval methods usually use self-training and clustering to generate pseudo labels. However, due to the large amount of noise in image and text data, the existing cross-modal target retrieval models do not perform well in cross-modal target retrieval tasks. In addition, because the clustering-based pseudo labels can only use the sample relationship of limited views, the reliability is poor, which further deteriorates the performance of the trained cross-modal target retrieval model. Therefore, how to obtain a cross-modal target retrieval model with better performance and achieve better target retrieval results is an important problem that needs to be solved urgently. Summary of the invention

[0004] The present invention provides a weakly supervised cross-modal target retrieval method based on multi-view relationship consistency, which is used to solve the defects of diversity, heterogeneity and complexity of multimodal data in the prior art and realize cross-modal target retrieval with high robustness.

[0005] The present invention provides a weakly supervised cross-modal object retrieval method based on multi-view relationship consistency, comprising the following steps:

[0006] Training cross-modal object retrieval models based on multimodal data;

[0007] Use the cross-modal target retrieval model to obtain cross-modal target retrieval results;

[0008] Among them, training a cross-modal target retrieval model based on multimodal data includes the following steps:

[0009] Extract visual embedding features based on sample images of the target domain of multimodal data; extract text embedding features based on text images of the target domain of multimodal data;

[0010] Based on the visual embedding features and text embedding features, a multi-view hypergraph is constructed through clustering method;

[0011] According to the multi-view hypergraph, the relationship consistency reasoning based on the label propagation algorithm is performed to obtain reliable pseudo labels;

[0012] The target retrieval model is obtained by training based on reliable pseudo labels.

[0013] According to the weakly supervised cross-modal target retrieval method based on multi-view relation consistency provided by the present invention, the relation consistency reasoning based on the label propagation algorithm is performed according to the multi-view hypergraph to obtain a reliable pseudo label, including the following steps:

[0014] Determine initial clustering and initial reliable pseudo labels based on the multi-view hypergraph;

[0015] Calculate the error matrix based on the initial clustering and initial pseudo-labels;

[0016] Label propagation is performed according to the error matrix to obtain reliable pseudo labels for all data instances.

[0017] According to the weakly supervised cross-modal target retrieval method based on multi-view relationship consistency provided by the present invention, the error matrix is ​​calculated according to the initial clustering and the initial pseudo-label, and its expression is:

[0018]

[0019] Among them, ε r is the error matrix of the reliable instance, ε u is the error matrix of the unreliable instance, is the initial prediction matrix, is a feature matrix consisting of reliable one-hot vectors and zeros at other positions;

[0020] Label propagation is performed according to the error matrix to obtain reliable pseudo labels for all data instances, which are expressed as:

[0021]

[0022] Where, the u subscript indicates the selection of items other than reliable instances in the error matrix, t is the number of iterations, ε is the error matrix, A is the adjacency matrix of the multi-view hypergraph, is the first preset coefficient, D is the degree matrix;

[0023] Get the corrected label prediction based on the error matrix Its expression is:

[0024]

[0025] Among them, s is the scale factor.

[0026] According to the weakly supervised cross-modal target retrieval method based on multi-view relationship consistency provided by the present invention, training is performed based on reliable pseudo labels to obtain a target retrieval model, specifically: training is performed based on an intra-modal matching objective function and a cross-modal matching objective function between visual or text embedding and ID prototypes to obtain a target retrieval model;

[0027] Total loss function The expression is:

[0028]

[0029] in, is the intra-modal matching objective function, is the cross-modal matching objective function.

[0030] According to the weakly supervised cross-modal target retrieval method based on multi-view relationship consistency provided by the present invention, the intra-modal matching objective function is expressed as:

[0031]

[0032] Among them, f i v is the i-th visual encoding, f i t is the i-th text encoding corresponding to the i-th visual encoding, c v represents the visual prototype, c v The subscript of is the identifier of the visual prototype, c t Represents the text prototype, c t The subscript is the identifier of the text prototype, τ is the double contrast loss, n t is the number of text IDs, n v is the number of visual IDs.

[0033] According to the weakly supervised cross-modal target retrieval method based on multi-view relationship consistency provided by the present invention, the cross-modal matching objective function is expressed as:

[0034]

[0035] Among them, f i v is the i-th visual encoding, f i t is the i-th text encoding corresponding to the i-th visual encoding, c v represents the visual prototype, c v The subscript of is the identifier of the visual prototype, c t Represents the text prototype, c t The subscript is the identifier of the text prototype, τ is the double contrast loss, n t is the number of text IDs, nv is the number of visual IDs.

[0036] The present invention also provides a weakly supervised cross-modal target retrieval device based on multi-view relationship consistency, comprising:

[0037] Model training module, used to train cross-modal target retrieval model based on multimodal data;

[0038] A cross-modal retrieval module, used to obtain cross-modal target retrieval results using a cross-modal target retrieval model;

[0039] Among them, training a cross-modal target retrieval model based on multimodal data includes the following steps:

[0040] Extract visual embedding features based on sample images of the target domain of multimodal data; extract text embedding features based on text images of the target domain of multimodal data;

[0041] Based on the visual embedding features and text embedding features, a multi-view hypergraph is constructed through clustering method;

[0042] According to the multi-view hypergraph, the relationship consistency reasoning based on the label propagation algorithm is performed to obtain reliable pseudo labels;

[0043] The target retrieval model is obtained by training based on reliable pseudo labels.

[0044] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a weakly supervised cross-modal target retrieval method based on multi-view relationship consistency as described above is implemented.

[0045] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described weakly supervised cross-modal target retrieval methods based on multi-view relationship consistency.

[0046] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned weakly supervised cross-modal target retrieval methods based on multi-view relationship consistency.

[0047] The weakly supervised cross-modal target retrieval method based on multi-view relationship consistency provided by the present invention can effectively fuse sample relations from multiple views by constructing a multi-view hypergraph and performing relationship consistency reasoning based on the label propagation algorithm according to the multi-view hypergraph, so as to make the information of multiple views complementary, and can mine the consistency in multi-view sample relations, realize multi-view-based training of weakly supervised cross-modal models, improve the robustness of the model, and thus obtain better cross-modal target retrieval performance for diverse, heterogeneous and complex multimodal data according to the weakly supervised cross-modal model. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 It is one of the flow charts of the weakly supervised cross-modal target retrieval method based on multi-view relationship consistency provided by the present invention;

[0050] Figure 2 This is the second flow chart of the weakly supervised cross-modal target retrieval method based on multi-view relationship consistency provided by the present invention;

[0051] Figure 3 It is a structural schematic diagram of a weakly supervised cross-modal target retrieval device based on multi-view relationship consistency provided by the present invention;

[0052] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] Combine the following Figure 1-Figure 4 A weakly supervised cross-modal object retrieval method based on multi-view relationship consistency according to an embodiment of the present invention is described.

[0055] Figure 1 is one of the flow charts of the weakly supervised cross-modal target retrieval method based on multi-view relationship consistency provided by the present invention, such as Figure 1As shown, step 110 and step 120 are included, which specifically include:

[0056] Step 110: training a cross-modal target retrieval model based on multimodal data;

[0057] Step 120: using the cross-modal target retrieval model to obtain a cross-modal target retrieval result;

[0058] Wherein, step 110 includes steps 111 to 114, which specifically include:

[0059] Step 111: extracting visual embedding features based on the target domain sample images of the multimodal data; extracting text embedding features based on the target domain text images of the multimodal data;

[0060] Step 112: construct a multi-view hypergraph by clustering method according to the visual embedding features and the text embedding features;

[0061] Step 113: Perform relation consistency reasoning based on the label propagation algorithm according to the multi-view hypergraph to obtain reliable pseudo labels;

[0062] Step 114: Perform training based on reliable pseudo labels to obtain a target retrieval model.

[0063] Figure 2 This is the second flow chart of the weakly supervised cross-modal target retrieval method based on multi-view relationship consistency according to an embodiment of the present invention. The steps of the weakly supervised cross-modal target retrieval method based on multi-view relationship consistency according to an embodiment of the present invention are specifically implemented as follows:

[0064] In step 111 of the embodiment of the present invention, the visual embedding features and the text embedding features may be features extracted using a text encoder and a visual encoder, or may be unsupervised manually designed features, such as TF-IDF. Specifically, in the embodiment of the present invention, let f θ and are the text encoder and the visual encoder, θ and as their parameters. is a given training data set, where I i is the i-th image, T i represents the i-th text description, and N is the number of text-image pairs. θ and Extract visual embedding features F respectively v and text embedding feature F t It is understandable that the embodiments of the present invention may use any text encoder and visual encoder to achieve the technical effects of the present invention.

[0065] In step 112 of the embodiment of the present invention, a multi-view hypergraph G is constructed by a clustering method based on the visual embedding features and the text embedding features. Specifically, a plurality of relationship graphs are first constructed. where g j = {V,E j} is the jth relationship graph, the image and text data X are the nodes of the relationship graph g, is the vertex set of the relationship graph, containing N nodes, is the edge set of the relationship graph, containing L edges, and M is the number of relationship graphs. k The instances corresponding to the included nodes belong to the same ID. Then multiple relationship graphs g are merged to obtain a multi-relationship hypergraph G = (V, E, W), where V is the vertex set of the multi-relationship hypergraph, is the set of hyperedges of a multi-relational hypergraph, and W is the edge weight of each hyperedge of the multi-relational hypergraph. In an embodiment of the present invention, in order to effectively model the sample relationships of the diversity and complexity of heterogeneous multimodal data, the embodiment of the present invention takes into account various factors, constructs a relationship graph as comprehensively as possible, and divides these factors into three categories, namely, features, metrics, and graphs. Features are used to represent samples, which can be extracted visual embeddings and text embeddings, or unsupervised hand-designed features, such as TF-IDF. The TF-IDF clustering graph is constructed using TF-IDF features, which emphasizes the co-occurrence frequency of words; metrics refer to similarity metrics used to measure feature similarity, such as cosine similarity and Jaccard distance; graphs represent how to construct a relationship graph, and the connections between nodes on the relationship graph are determined based on the previous sample similarity. In an embodiment of the present invention, a relationship graph can be constructed by kNN, which takes into account the nearest neighbor relationship; a relationship graph can also be constructed by a clustering method based on neighborhood density.

[0066] The edges of the clustering relationship graph obtained in step 111 After merging, we can get the multi-view hypergraph G, which can be represented by an association matrix express:

[0067]

[0068] The weight of each hyperedge e is:

[0069]

[0070] Where S(i,j) represents the similarity between node i and node j in hyperedge e, and σ is the median similarity of all vertex pairs.

[0071] The adjacency matrix of the multi-view hypergraph G is:

[0072]

[0073] in is the corresponding diagonal matrix of the hypergraph degree. Using A, we can propagate labels on the graph and optimize pseudo labels.

[0074] Through the above steps, the embodiment of the present invention enables multiple relationship graphs to complement each other and provides diverse perspectives of sample relationships. That is, the multi-view hypergraph G can be used to mine complementary information in different views and characterize complex sample relationships.

[0075] In the embodiment of the present invention, step 113 includes the following steps:

[0076] Determine initial clustering and initial reliable pseudo labels based on the multi-view hypergraph;

[0077] Calculate the error matrix based on the initial clustering and initial pseudo-labels;

[0078] Label propagation is performed according to the error matrix to obtain reliable pseudo labels for all data instances.

[0079] Specifically, first, according to the relationship graph g in the multi-view hypergraph joint The sample relationship on N c Initial clusters and pseudo labels y joint The first k instances closest to the cluster center in each cluster are selected as initial reliable pseudo labels for propagation, and then the initial outliers generated by the clustering are regarded as unreliable instances.

[0080] Get k reliable instances X in each cluster r and its corresponding pseudo label y r After that, the error matrix can be calculated, and its expression is:

[0081]

[0082] Among them, ε r is the error matrix of the reliable instance, ε u is the error matrix of the unreliable instance, is the initial prediction matrix, is a feature matrix consisting of reliable one-hot vectors and zeros at other positions,

[0083] Then the residual error is propagated, and the label is propagated according to the error matrix to obtain reliable pseudo labels for all data instances, which are expressed as:

[0084]

[0085] Where, the u subscript indicates the selection of items other than reliable instances in the error matrix, t is the number of iterations, ε is the error matrix, A is the adjacency matrix of the multi-view hypergraph, is the first preset coefficient, D is the degree matrix;

[0086] Get the corrected label prediction based on the error matrix Its expression is:

[0087]

[0088] Among them, s is the scale factor.

[0089] The prediction is smoothed by label propagation, which is expressed as:

[0090]

[0091] in is the second preset coefficient, is the prediction matrix, and the superscript is the number of iterations;

[0092] Iteratively updating the predictions converges to the final predictions, resulting in reliable pseudo-labels for all data instances.

[0093] In the embodiment of the present invention, the first preset coefficient and the second preset coefficient are 0.99 and 0.9 respectively.

[0094] The embodiment of the present invention performs label propagation through the above steps, and can adaptively fuse complementary information from multiple views to generate a final pseudo label for learning the joint feature embedding space of visual text. Therefore, even with the diversity, heterogeneity and complexity of multimodal data, the embodiment of the present invention can also obtain better weakly supervised cross-modal target retrieval performance.

[0095] Furthermore, in an embodiment of the present invention, the text encoder and the visual encoder can be optimized according to the final optimized pseudo-labels obtained through training to improve the performance of the text encoder and the visual encoder in extracting embedded features.

[0096] The embodiment of the present invention maintains two ID-aware dictionaries, which store feature vectors and As a visual and textual ID prototype, where n v and n t is the number of IDs, and Representing the visual prototype and text prototype of the i-th instance, and performing model training based on two ID-aware dictionaries. Specifically, in the embodiment of the present invention, step 114 is specifically: training based on the intra-modal matching objective function and the cross-modal matching objective function between the visual or text embedding and the ID prototype to obtain a target retrieval model;

[0097] Total loss function The expression is:

[0098]

[0099] in, is the intra-modal matching objective function, is the cross-modal matching objective function.

[0100] Intra-modal matching objective function The cross-modal matching objective function is expressed as:

[0101]

[0102] Among them, f i v is the i-th visual encoding, f i t is the i-th text encoding corresponding to the i-th visual encoding, c v represents the visual prototype, c v The subscript of is the identifier of the visual prototype, c t Represents the text prototype, c t The subscript is the identifier of the text prototype, τ is the double contrast loss, n t is the number of text IDs, n v is the number of visual IDs.

[0103] Cross-modal matching objective function The expression is:

[0104]

[0105] Among them, f i v is the i-th visual encoding, f i t is the i-th text encoding corresponding to the i-th visual encoding, c v represents the visual prototype, c v The subscript of is the identifier of the visual prototype, c t Represents the text prototype, c t The subscript is the identifier of the text prototype, τ is the double contrast loss, n t is the number of text IDs, n v is the number of visual IDs.

[0106] The embodiment of the present invention uses the above loss function design to enable the cross-modal target retrieval model to learn a discriminative visual-text joint feature space. The value of the loss function approaches 0 when it is similar to the ID prototype corresponding to it and dissimilar to other non-corresponding prototypes.

[0107] To summarize, the weakly supervised cross-modal target retrieval method based on multi-view relationship consistency of the embodiment of the present invention, by constructing a multi-view hypergraph and performing relationship consistency reasoning based on the label propagation algorithm according to the multi-view hypergraph, can effectively integrate sample relations from multiple views, make the information of multiple views complementary, and can mine the consistency in multi-view sample relations, realize multi-view-based training of weakly supervised cross-modal models, improve the robustness of the model, and thus obtain better cross-modal target retrieval performance for diverse, heterogeneous and complex multimodal data according to the weakly supervised cross-modal model.

[0108] The weakly supervised cross-modal target retrieval device based on multi-view relationship consistency provided by the present invention is described below. The weakly supervised cross-modal target retrieval device based on multi-view relationship consistency described below and the weakly supervised cross-modal target retrieval method based on multi-view relationship consistency described above can be referenced to each other.

[0109] Figure 3 is a schematic diagram of the structure of a weakly supervised cross-modal target retrieval device based on multi-view relationship consistency provided by the present invention, such as Figure 3 As shown, including:

[0110] A model training module 310, for training a cross-modal target retrieval model based on multimodal data;

[0111] A cross-modal retrieval module 320, configured to obtain a cross-modal target retrieval result using a cross-modal target retrieval model;

[0112] Among them, training a cross-modal target retrieval model based on multimodal data includes the following steps:

[0113] Extract visual embedding features based on sample images of the target domain of multimodal data; extract text embedding features based on text images of the target domain of multimodal data;

[0114] Based on the visual embedding features and text embedding features, a multi-view hypergraph is constructed through clustering method;

[0115] According to the multi-view hypergraph, the relationship consistency reasoning based on the label propagation algorithm is performed to obtain reliable pseudo labels;

[0116] The target retrieval model is obtained by training based on reliable pseudo labels.

[0117] It can be seen that the weakly supervised cross-modal target retrieval device based on multi-view relationship consistency of the embodiment of the present invention can construct a multi-view hypergraph, and perform relationship consistency reasoning based on the label propagation algorithm according to the multi-view hypergraph, can effectively integrate sample relations from multiple views, make the information of multiple views complementary, and can mine the consistency in multi-view sample relations, realize weakly supervised cross-modal model based on multi-view training, improve the robustness of the model, and thus obtain better cross-modal target retrieval performance for diverse, heterogeneous and complex multimodal data according to the weakly supervised cross-modal model.

[0118] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute a weakly supervised cross-modal target retrieval method based on multi-view relationship consistency, including the following steps:

[0119] Training cross-modal object retrieval models based on multimodal data;

[0120] Use the cross-modal target retrieval model to obtain cross-modal target retrieval results;

[0121] Among them, training a cross-modal target retrieval model based on multimodal data includes the following steps:

[0122] Extract visual embedding features based on sample images of the target domain of multimodal data; extract text embedding features based on text images of the target domain of multimodal data;

[0123] Based on the visual embedding features and text embedding features, a multi-view hypergraph is constructed through clustering method;

[0124] According to the multi-view hypergraph, the relationship consistency reasoning based on the label propagation algorithm is performed to obtain reliable pseudo labels;

[0125] The target retrieval model is obtained by training based on reliable pseudo labels.

[0126] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0127] On the other hand, the present invention also provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer can execute the weakly supervised cross-modal target retrieval method based on multi-view relationship consistency provided by the above methods, including the following steps:

[0128] Training cross-modal object retrieval models based on multimodal data;

[0129] Use the cross-modal target retrieval model to obtain cross-modal target retrieval results;

[0130] Among them, training a cross-modal target retrieval model based on multimodal data includes the following steps:

[0131] Extract visual embedding features based on sample images of the target domain of multimodal data; extract text embedding features based on text images of the target domain of multimodal data;

[0132] Based on the visual embedding features and text embedding features, a multi-view hypergraph is constructed through clustering method;

[0133] According to the multi-view hypergraph, the relationship consistency reasoning based on the label propagation algorithm is performed to obtain reliable pseudo labels;

[0134] The target retrieval model is obtained by training based on reliable pseudo labels.

[0135] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the weakly supervised cross-modal target retrieval method based on multi-view relationship consistency provided by the above methods, comprising the following steps:

[0136] Training cross-modal object retrieval models based on multimodal data;

[0137] Use the cross-modal target retrieval model to obtain cross-modal target retrieval results;

[0138] Among them, training a cross-modal target retrieval model based on multimodal data includes the following steps:

[0139] Extract visual embedding features based on sample images of the target domain of multimodal data; extract text embedding features based on text images of the target domain of multimodal data;

[0140] Based on the visual embedding features and text embedding features, a multi-view hypergraph is constructed through clustering method;

[0141] According to the multi-view hypergraph, the relationship consistency reasoning based on the label propagation algorithm is performed to obtain reliable pseudo labels;

[0142] The target retrieval model is obtained by training based on reliable pseudo labels.

[0143] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0144] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A weakly supervised cross-modal object retrieval method based on multi-view relation consistency, characterized in that: The following steps are involved: Training cross-modal object retrieval models based on multimodal data; Use the cross-modal target retrieval model to obtain cross-modal target retrieval results; Among them, training a cross-modal target retrieval model based on multimodal data includes the following steps: Extract visual embedding features based on sample images of the target domain of multimodal data; extract text embedding features based on text images of the target domain of multimodal data; Based on the visual embedding features and text embedding features, a multi-view hypergraph is constructed through clustering method; According to the multi-view hypergraph, the relationship consistency reasoning based on the label propagation algorithm is performed to obtain reliable pseudo labels; Train based on reliable pseudo labels to obtain a target retrieval model; The method of performing relation consistency reasoning based on a label propagation algorithm according to a multi-view hypergraph to obtain a reliable pseudo label includes the following steps: Determine initial clustering and initial reliable pseudo labels based on the multi-view hypergraph; Calculate the error matrix based on the initial clustering and initial pseudo-labels; Perform label propagation based on the error matrix to obtain reliable pseudo labels for all data instances; According to the initial clustering and initial pseudo-label, the error matrix is ​​calculated, which is expressed as: Among them, ε r is the error matrix of the reliable instance, ε u is the error matrix of the unreliable instance, is the initial prediction matrix, is a feature matrix consisting of reliable one-hot vectors and zeros at other positions; Label propagation is performed according to the error matrix to obtain reliable pseudo labels for all data instances, which are expressed as: Where, the u subscript indicates the selection of items other than reliable instances in the error matrix, t is the number of iterations, ε is the error matrix, A is the adjacency matrix of the multi-view hypergraph, is the first preset coefficient, D is the degree matrix; Get the corrected label prediction based on the error matrix Its expression is: Among them, s is the scale coefficient; The prediction is smoothed by label propagation, which is expressed as: in is the second preset coefficient, is the prediction matrix, and the superscript is the number of iterations; Iteratively update the predictions until they converge to the final predictions, obtaining reliable pseudo-labels for all data instances; The target retrieval model is obtained by training based on reliable pseudo labels, specifically: the target retrieval model is obtained by training based on the intra-modal matching objective function and the cross-modal matching objective function between the visual or text embedding and the ID prototype; Total loss function The expression is: in, is the intra-modal matching objective function, is the cross-modal matching objective function; The intra-modal matching objective function is expressed as: Among them, f i v is the i-th visual encoding, f i t is the i-th text encoding corresponding to the i-th visual encoding, c v represents the visual prototype, c v The subscript of is the identifier of the visual prototype, c t Represents the text prototype, c t The subscript is the identifier of the text prototype, τ is the double contrast loss, n t is the number of text IDs, n v is the number of visual IDs; The cross-modal matching objective function is expressed as: Among them, f i v is the i-th visual encoding, f i t is the i-th text encoding corresponding to the i-th visual encoding, c v represents the visual prototype, c v The subscript of is the identifier of the visual prototype, c t Represents the text prototype, c t The subscript is the identifier of the text prototype, τ is the double contrast loss, n t is the number of text IDs, n v is the number of visual IDs.

2. A weakly supervised cross-modal object retrieval device based on multi-view relation consistency, characterized in that: include: Model training module, used to train cross-modal target retrieval model based on multimodal data; A cross-modal retrieval module, used to obtain cross-modal target retrieval results using a cross-modal target retrieval model; Among them, training a cross-modal target retrieval model based on multimodal data includes the following steps: Extract visual embedding features based on sample images of the target domain of multimodal data; extract text embedding features based on text images of the target domain of multimodal data; Based on the visual embedding features and text embedding features, a multi-view hypergraph is constructed through clustering method; According to the multi-view hypergraph, the relationship consistency reasoning based on the label propagation algorithm is performed to obtain reliable pseudo labels; Train based on reliable pseudo labels to obtain a target retrieval model; The method of performing relation consistency reasoning based on a label propagation algorithm according to a multi-view hypergraph to obtain a reliable pseudo label includes the following steps: Determine initial clustering and initial reliable pseudo labels based on the multi-view hypergraph; Calculate the error matrix based on the initial clustering and initial pseudo-labels; Perform label propagation based on the error matrix to obtain reliable pseudo labels for all data instances; According to the initial clustering and initial pseudo-label, the error matrix is ​​calculated, which is expressed as: Among them, ε r is the error matrix of the reliable instance, ε u is the error matrix of the unreliable instance, is the initial prediction matrix, is a feature matrix consisting of reliable one-hot vectors and zeros at other positions; Label propagation is performed according to the error matrix to obtain reliable pseudo labels for all data instances, which are expressed as: Where, the u subscript indicates the selection of items other than reliable instances in the error matrix, t is the number of iterations, ε is the error matrix, A is the adjacency matrix of the multi-view hypergraph, is the first preset coefficient, D is the degree matrix; Get the corrected label prediction based on the error matrix Its expression is: Among them, s is the scale coefficient; The prediction is smoothed by label propagation, which is expressed as: in is the second preset coefficient, is the prediction matrix, and the superscript is the number of iterations; Iteratively update the predictions until they converge to the final predictions, obtaining reliable pseudo-labels for all data instances; The target retrieval model is obtained by training based on reliable pseudo labels, specifically: the target retrieval model is obtained by training based on the intra-modal matching objective function and the cross-modal matching objective function between the visual or text embedding and the ID prototype; Total loss function The expression is: in, is the intra-modal matching objective function, is the cross-modal matching objective function; The intra-modal matching objective function is expressed as: Among them, f i v is the i-th visual encoding, f i t is the i-th text encoding corresponding to the i-th visual encoding, c v represents the visual prototype, c v The subscript of is the identifier of the visual prototype, c t Represents the text prototype, c t The subscript is the identifier of the text prototype, τ is the double contrast loss, n t is the number of text IDs, n v is the number of visual IDs; The cross-modal matching objective function is expressed as: Among them, f i v is the i-th visual encoding, f i t is the i-th text encoding corresponding to the i-th visual encoding, c v represents the visual prototype, c v The subscript of is the identifier of the visual prototype, c t Represents the text prototype, c t The subscript is the identifier of the text prototype, τ is the double contrast loss, n t is the number of text IDs, n v is the number of visual IDs.

3. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the weakly supervised cross-modal target retrieval method based on multi-view relationship consistency as described in claim 1 is implemented.

4. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the weakly supervised cross-modal target retrieval method based on multi-view relationship consistency as claimed in claim 1 is implemented.

5. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the weakly supervised cross-modal target retrieval method based on multi-view relationship consistency as claimed in claim 1 is implemented.

Images