Training and retrieval methods for cross-modal hashing models with partially missing labels

Abstract

This application discloses a training and retrieval method for a cross-modal hashing model that addresses partially missing labels. The training method constructs a learnable preset label prompt to indicate label completeness. A label prompt vector is constructed using the preset label prompt and the incomplete labels of cross-modal training samples. The preset label prompt is then learned using the completeness score of the label prompt vector. Finally, the incomplete labels of the cross-modal training samples are recovered based on the learned target label prompt to effectively perceive missing labels, reduce the impact of incomplete labels on the cross-modal hashing model, and improve the model performance of the cross-modal hashing model. Furthermore, the complementary semantic propagation and adaptive negative pairing strategies are used to reduce unknown pairs to balance the distribution of positive and negative pairs, thereby further improving the model performance of the cross-modal hashing model.

Description

Technical Field

[0001] The present application relates to the field of information science and technology, and in particular to a training and retrieval method for a cross-modal hash model that deals with partial label loss. Background Art

[0002] Cross-modal hashing (Cross-modal hashing) compresses multimodal data (e.g., images and text) into compact binary codes while maintaining their cross-modal semantic similarity across high-dimensional data points. The spatiotemporal efficiency of hashing algorithms has been widely recognized and has been widely used in search engines and recommendation systems that require fast indexing and retrieval of instances from different modalities. However, existing cross-modal hashing retrieval requires complete annotation of the entire dataset. When the dataset expands rapidly, this expensive supervision is difficult to implement due to limited labor resources, making it difficult for existing cross-modal hashing to address the core problem of missing labels that arises in reality and remains unresolved.

[0003] In the case of partial missing labels, the sample data is annotated with incomplete labels, which include some explicitly annotated semantic labels and some unknown labels that are annotated with "unknown". The unknown labels will fundamentally destroy the pairwise relationship, especially for cross-modal learning, and cross-modal hashing with incomplete labels will inevitably encounter similarity learning that is destroyed due to another type of uncertainty, making the modal retrieval model vulnerable to incomplete labels, thereby affecting the accuracy of cross-modal hashing retrieval.

[0004] Therefore existing technology still needs to be improved and improved. Summary of the Invention

[0005] The technical problem to be solved by this application is to provide a training and retrieval method for a cross-modal hash model that can cope with partial label missing in response to the shortcomings of the existing technology.

[0006] In order to solve the above technical problems, the first aspect of the present application provides a training method for a cross-modal hash model for partially missing labels, wherein the training method for a cross-modal hash model for partially missing labels specifically includes:

[0007] For each training sample in the training sample set, an initial label set is randomly selected from the positive semantic labels in the incomplete labels of the training sample, a plurality of reference label sets are generated based on the initial label set and the incomplete labels, and a label prompt vector corresponding to the initial label set and a reference label prompt vector corresponding to each reference label set are constructed according to a preset label prompt, wherein the reference label set includes the positive semantic labels in the initial label set, or includes the positive semantic labels in the initial label set and the negative semantic labels in the incomplete labels;

[0008] Calculating the first completeness score of the label hint vector of each training sample and the second completeness score of each control label hint vector;

[0009] constructing a total contrast loss term based on all first complete scores and all second complete scores, and learning the preset label prompt based on the total contrast loss term to obtain a target label prompt;

[0010] Based on the target label prompt, the unknown semantic classes in the incomplete labels of each training sample are recovered by using the semantic label set corresponding to the training sample set to obtain the recovered labels of each training sample, so as to obtain the recovered training sample set;

[0011] A preset hash model is trained based on the restored training sample set to obtain a cross-modal hash model.

[0012] The training method for a cross-modal hashing model with partially missing labels, wherein generating a plurality of control label sets based on the initial label set and the incomplete label specifically includes:

[0013] Performing at least one adjustment operation on the initial label set to obtain a control label set, wherein the adjustment operation is a random addition of negative semantic labels and / or a random deletion of positive semantic labels;

[0014] Repeating at least one adjustment operation on the initial label set until a plurality of control label sets are obtained.

[0015] The training method for the cross-modal hashing model with partially missing labels, wherein the preset label prompt is:

[0016]

[0017]

[0018]

[0019]

[0020] Among them, P crepresents the learnable prompt for the cth semantic label, P head represents an agnostic prefix of a learnable semantic label, P tail Indicates the agnostic suffix of the learnable semantic label, C desc represents the description label set in the preset label prompt, σ(·) represents the permutation on the description label set, All represent learnable parameters, m, n, m a and n a All are positive integers.

[0021] The training method for a cross-modal hashing model for partially missing labels, wherein the first completeness score and the second completeness score are both obtained based on a completeness score calculation formula, wherein the completeness score calculation formula is:

[0022] Φ(K)=E t (P(K)) Τ h / τ

[0023] Among them, K represents the label set, E t (·) represents the text encoder, P(K) represents the label prompt vector generated based on the label set K, τ represents the temperature parameter, and h represents the modal feature corresponding to the training sample.

[0024] The training method for a cross-modal hashing model with partially missing labels, wherein, based on the target label prompt, the label recovery of the unknown semantic classes in the incomplete labels of each training sample is performed using the semantic label set corresponding to the training sample set to obtain the recovered labels of each training sample, so as to obtain the recovered training sample set specifically includes:

[0025] For each training sample in the training sample set, determining all positive semantic labels and all unknown semantic labels corresponding to the training sample based on the semantic label set of the training sample set and the incomplete label of the training sample;

[0026] With the goal of achieving the highest completeness score, the restored labels of the unknown semantic classes in the incomplete labels of the training samples are searched in the semantic label set consisting of all positive semantic labels and all unknown semantic labels to obtain the restored labels corresponding to the training samples;

[0027] The restored label of each training sample is used to replace its corresponding incomplete label to obtain a restored training sample set.

[0028] The training method for a cross-modal hash model for partially missing labels, wherein the training of a preset hash model based on a restored training sample set to obtain a cross-modal hash model specifically includes:

[0029] For each training batch in the restored training sample set, obtain the modal features of each single modal data in the training samples in the training batch;

[0030] According to the restored labels of each training sample in the training batch, the complementary scores between each two training samples are calculated to obtain the complementary score matrix;

[0031] For each training sample in the training batch, determining a complementary training sample corresponding to the training sample according to the complementary score matrix, fusing the modal feature of the training sample with the model features of all complementary training samples to obtain an updated modal feature, and fusing the restored training sample with the restored all complementary training samples to obtain an updated label;

[0032] Based on the updated modal features and updated labels of each training sample, a loss function term is constructed, and a preset hash model is trained based on the loss function term to obtain a cross-modal hash model.

[0033] In the training method for the cross-modal hashing model with partially missing labels, the similarity matrix used in constructing the loss function term is:

[0034]

[0035]

[0036] Among them, S∈([0,1]∪{u}) B*B represents the initial similarity matrix, r represents s ij =0 and s ij >0, B represents the number of samples in the training batch, represents the updated label of the i-th training sample, The updated label of the jth training sample, C represents the semantic label set corresponding to the restored training sample set, Used to perform 0-1 masking, setting unknown elements to 1 and other elements to 0, R∈{0,1} B×B represents a random mask, represents the Hadamard product.

[0037] A second aspect of the present application provides a cross-modal hash retrieval method for partially missing labels, using a cross-modal hash model trained using the above-described training method for a cross-modal hash model for partially missing labels. The cross-modal hash retrieval method for partially missing labels specifically includes:

[0038] receiving unimodal data to be retrieved, and determining a hash code of the unimodal data to be retrieved using the cross-modal hash model;

[0039] According to the similarity between the hash code and each modal hash code in the modal hash code library corresponding to the preset modal database, a preset number of candidate modal data corresponding to the single modal data to be retrieved are selected from the preset modal database.

[0040] A third aspect of the present application provides a training device for a cross-modal hash model with partially missing labels, wherein the training device for a cross-modal hash model with partially missing labels specifically includes:

[0041] a generation module configured to randomly select, for each training sample in a training sample set, an initial label set from the positive semantic labels in the incomplete labels of the training sample, generate a plurality of reference label sets based on the initial label set and the incomplete labels, and construct a label prompt vector corresponding to the initial label set and a reference label prompt vector corresponding to each reference label set according to a preset label prompt, wherein the reference label set includes the positive semantic labels in the initial label set, or includes the positive semantic labels in the initial label set and the negative semantic labels in the incomplete labels;

[0042] a calculation module, configured to calculate a first completeness score of the label hint vector of each training sample and a second completeness score of each control label hint vector;

[0043] a learning module, configured to construct a total contrast loss term based on all first completeness scores and all second completeness scores, and learn the preset label prompt based on the total contrast loss term to obtain a target label prompt;

[0044] a recovery module, configured to recover the incomplete label of each training sample based on the target label prompt and the semantic label set corresponding to the training sample set to obtain a recovered label of each training sample, and replace the incomplete label of each training sample with the recovered label of each training sample to obtain a recovered training sample set;

[0045] The training module is used to train a preset hash model based on the restored training sample set to obtain a cross-modal hash model.

[0046] A fourth aspect of the present application provides a terminal device, comprising: a processor and a memory;

[0047] The memory stores a computer-readable program executable by the processor;

[0048] When the processor executes the computer-readable program, it implements the steps of the training method of the cross-modal hash model for dealing with partial missing labels as described above, and / or implements the steps of the cross-modal hash retrieval method for dealing with partial missing labels as described above.

[0049] Beneficial effects: Compared with the prior art, the present application provides a training and retrieval method for a cross-modal hash model that deals with partially missing labels. The training method constructs a learnable preset label prompt, builds a label prompt vector through the preset label prompt and the incomplete labels of the cross-modal training samples, and then learns the preset label prompt through the completeness score of the label prompt vector. Finally, based on the learned target label prompt, the incomplete labels of the cross-modal training samples are restored to effectively perceive the missing labels, reduce the impact of incomplete labels on the cross-modal hash model, and improve the model performance of the cross-modal hash model. At the same time, the complementary semantic propagation and adaptive negative pairing strategy are used to reduce unknown pairs to balance the positive-negative pair distribution, thereby further improving the model performance of the cross-modal hash model. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0051] Figure 1 A flowchart of a method for training a cross-modal hash model with partially missing labels provided in an embodiment of the present application.

[0052] Figure 2 A flowchart of a specific implementation method of the training method of the cross-modal hash model for partially missing labels provided in an embodiment of the present application.

[0053] Figure 3 Schematic diagram of the principle of the control label set construction process.

[0054] Figure 4 A schematic diagram of the structural principles of a training device for a cross-modal hashing model with partially missing labels, provided in an embodiment of the present application.

[0055] Figure 5 This is a schematic diagram of the structure of the terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] The embodiments of the present application provide a training and retrieval method for a cross-modal hash model that addresses partial label loss. To make the objectives, technical solutions, and effects of the present application clearer and more explicit, the present application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to explain the present application and are not intended to limit the present application.

[0057] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0058] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0059] It should be understood that the sequence numbers and sizes of the steps in this embodiment do not imply the order of execution. The order of execution of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of this application.

[0060] The application content will be further explained below through description of embodiments in conjunction with the accompanying drawings.

[0061] This embodiment provides a training method for a cross-modal hashing model with partially missing labels, such as Figure 1 As shown, the method includes:

[0062] S10. For each training sample in the training sample set, randomly select an initial label set from the positive semantic labels in the incomplete labels of the training sample, generate several control label sets based on the initial label set and the incomplete labels, and construct a label prompt vector corresponding to the initial label set and a control label prompt vector corresponding to each control label set according to a preset label prompt.

[0063] Specifically, the training sample set includes several cross-modal training samples, each of which includes several unimodal data and incomplete labels. The unimodal data can be text modal data, image modal data, speech modal data, and video modal data, etc. The incomplete labels include several positive semantic labels, several negative semantic labels, and several unknown semantic labels. For example, taking several unimodal data including text modal data and image modal data as an example, the training sample set can be expressed as:

[0064]

[0065] Among them, f i represents the image modality data of the i-th training sample, g i represents the text modality data of the i-th training sample, l i Represents the incomplete label of the i-th training sample, which is the semantic label matrix L∈{1, 0, u} corresponding to the training dataset C×N In the i-th column, N represents the number of samples in the training sample set, C represents the number of categories of semantic categories, u represents unknown semantic labels, 1 represents positive semantic labels, and 0 represents negative semantic labels.

[0066] Furthermore, the initial tag set may include all positive semantic tags in the incomplete tags, or may include some positive semantic tags in the incomplete tags. In the embodiment of the present application, the initial tag set includes some positive semantic tags in the incomplete tags. For example, the positive semantic tag set consisting of all positive semantic tags in the incomplete tags is recorded as All negative semantic label sets are denoted as Initial label set for A non-empty subset of the positive semantic label set. The embodiment of the present application constructs a label prompt vector by adopting a non-empty subset of the positive semantic label set. When adopting the positive semantic label set, directly maximizing the consistency between the label prompt and the modal representation will cause deviation, making it impossible to capture its integrity level, resulting in invalid semantic recovery, thereby improving the accuracy of the recovered labels obtained subsequently.

[0067] In one implementation, the comparison tag set includes the positive semantic tags in the initial tag set, or includes the positive semantic tags in the initial tag set and the negative semantic tags in the incomplete tags. Based on this, generating a plurality of comparison tag sets based on the initial tag set and the incomplete tags specifically includes:

[0068] Performing at least one adjustment operation on the initial label set to obtain a control label set, wherein the adjustment operation is a random addition of negative semantic labels and / or a random deletion of positive semantic labels;

[0069] Repeating at least one adjustment operation on the initial label set until a plurality of control label sets are obtained.

[0070] Specifically, the adjustment operation is a random addition of negative semantic labels, a random deletion of positive semantic labels, and an adjustment operation consisting of a random addition of negative semantic labels and a random deletion of positive semantic labels. By performing the three adjustment operations on the initial label set, three types of control label sets can be constructed to introduce pairwise relationships with the initial positive semantic labels. Figure 2 As shown, the plurality of control label sets include at least three control label sets, one of the at least three control label sets being obtained by performing a random addition of negative semantic labels, one being obtained by performing a random deletion of positive semantic labels, one being obtained by performing a random addition of negative semantic labels, and one being obtained by performing a random addition of negative semantic labels and a random deletion of positive semantic labels. Of course, in practical applications, the initial positive semantic labels may be subjected to multiple random deletion of positive semantic labels, or multiple random addition of negative semantic labels, or multiple random deletion of positive semantic labels and multiple random addition of negative semantic labels.

[0071] like Figure 3 As shown, 1), delete the class (that is, the initial label set Perform random deletion of positive semantic labels) on the initial label set By performing random deletion of positive semantic labels, we can obtain a control label set. 2) Join Negative semantic labels in the initial label set (i.e. Perform the random addition of negative semantic labels operation) to obtain the control label set 3) Use Negative semantic label replacement in the initial label set Randomly delete positive semantic labels and randomly add negative semantic labels) to obtain the control label set in, and In addition, it should be noted that generating negative sets by replacement is equivalent to continuously performing deletion and addition, which can improve the robustness of the model.

[0072] In one implementation, the preset label prompt is pre-built and is a learnable prompt template, which can be expressed as:

[0073]

[0074]

[0075]

[0076]

[0077] Among them, P c represents the learnable prompt for the cth semantic label, P head represents an agnostic prefix of a learnable semantic label, P tail Indicates the agnostic suffix of the learnable semantic label, C desc represents the description label set in the preset label prompt, σ(·) represents the permutation on the description label set, All represent learnable parameters, m, n, m a and n a All are positive integers.

[0078] In the implementation of the present application, by adopting the above-mentioned expression method, by defining it in the context of multiple semantic categories, the complexity between classes and the completeness of the description of the entire label can be captured, and the label information used for semantic recovery can be used for knowledge, thereby improving the accuracy of label recovery.

[0079] Of course, in actual applications, the preset label prompt may also be expressed in other ways. For example, the preset label prompt may be expressed as follows:

[0080]

[0081]

[0082] Among them, N pre represents the number of learnable tokens, and CN represents the class name of the c-th class.

[0083] S20: Calculate a first completeness score of the label hint vector of each training sample and a second completeness score of each control label hint vector.

[0084] Specifically, the first completeness score is used to describe the label completeness of the label hint vector, and the second completeness score is used to describe the label completeness of the control label hint vector, wherein the first completeness score and the second completeness score can be obtained by extracting the encoding features by feature encoding the various corresponding label hint vectors, and then determining based on the encoding features, or can be determined based on the encoding features and the modal features of a single modal data in the training sample, or can be determined based on the encoding features and the modal features of all single modal data in the training sample. For example, the ratio of the encoding feature to the temperature parameter of the encoder used to determine the encoding feature is directly used as the completeness score, or the ratio of the inner product of the encoding feature and the modal feature vector of the single modal data and the temperature parameter is used as the completeness score, etc.

[0085] In a specific implementation, the first completeness score and the second completeness score are both obtained based on a completeness score calculation formula, where the completeness score calculation formula is:

[0086] Φ(K)=E t (P(K)) Τ h / τ

[0087] Among them, K represents the label set, E t (·) represents the text encoder, P(K) represents the label hint vector generated based on the label set K, τ represents the temperature parameter, and h represents the modal feature corresponding to the training sample. It should be noted that h can be the modal feature of a single modal data of the training sample, or it can be the fusion feature of the modal features of all single modal data. For example, if the training sample includes text modal data and image modal data, h can be the modal feature of the text modal data and the modal feature of the image modal data weighted together.

[0088] S30: constructing a total contrast loss term based on all first complete scores and all second complete scores, and learning the preset label prompt based on the total contrast loss term to obtain a target label prompt.

[0089] Specifically, the total contrast loss term includes the contrast loss between the label prompt vector corresponding to each training sample and the corresponding label prompt vectors of each control. That is, the label prompt vector corresponding to each training sample and each control label prompt vector form a pairwise relationship, and then the contrast loss of each pairwise relationship is calculated separately to obtain the sample contrast loss corresponding to the training sample. Then, the sample contrast losses of all training samples are added together to obtain the total contrast loss term. Based on this, the total contrast loss term can be expressed as:

[0090]

[0091]

[0092] in, Represents the total contrast loss term, N represents the number of samples in the training sample set, represents the positive semantic label set of the i-th training sample, represents the initial label set, represents the negative semantic label set of the i-th training sample, Both represent the control label set, and m represents the boundary hyperparameter separating different levels of completeness.

[0093] S40 , based on the target label prompt, recover the unknown semantic classes in the incomplete labels of each training sample using the semantic label set corresponding to the training sample set to obtain a recovered training sample set.

[0094] Specifically, after obtaining the target label prompt, all positive semantic labels in the incomplete labels of the training sample can be used to form a positive semantic label set as the original semantic label set, and then a control semantic label set is generated by adding semantic labels to the unknown semantic classes in the incomplete labels. Then, an original label prompt vector is formed based on the target label prompt and the original semantic label set, and a control label prompt vector is generated based on the control semantic label set. Then, the original complete score is determined according to the original label prompt vector, and the control complete score is determined based on the control label prompt vector. Then, based on the original complete score as a constraint condition and the maximum control complete score as the target, the recovery label of the unknown semantic class in the incomplete labels of the training sample is searched.

[0095] Based on this, in one implementation, based on the target label prompt, recovering the unknown semantic classes in the incomplete labels of each training sample using the semantic label set corresponding to the training sample set to obtain the recovered training sample set specifically includes:

[0096] For each training sample in the training sample set, determining all positive semantic labels and all unknown semantic labels corresponding to the training sample based on the semantic label set of the training sample set and the incomplete label of the training sample;

[0097] With the goal of achieving the highest completeness score, the restored labels of the unknown semantic classes in the incomplete labels of the training samples are searched in the semantic label set consisting of all positive semantic labels and all unknown semantic labels to obtain the restored labels corresponding to the training samples;

[0098] The restored label of each training sample is used to replace its corresponding incomplete label to obtain a restored training sample set.

[0099] Specifically, all positive semantic labels refer to all positive semantic labels included in the incomplete labels, and the unknown semantic labels include the semantic labels corresponding to all unknown semantic classes in the incomplete labels. It is understood that after obtaining the semantic label set corresponding to the training sample set, all semantic labels included in the incomplete labels are removed from the semantic label set to obtain all unknown semantic labels corresponding to the training sample, where all semantic labels in the incomplete labels include both positive and negative semantic labels.

[0100] After obtaining all positive semantic labels and all unknown semantic labels, the positive semantic label set composed of all positive semantic labels can be used as the original label set, and the completeness score corresponding to the positive semantic label set can be used as a constraint condition. Then, the highest completeness score is set as the goal, so as to use recursive search in In the process of positive semantic label detection, the labels of unknown semantic classes are recovered. At the same time, the positive classes found are merged into To restore the label, is a currently unknown semantic class, and The monotonically increasing hint completeness is ensured until this inequality fails, and the search ends with a positive score φ = Φ i (Q) Stop, where Q is the recovered positive semantic label set. In addition, it can be further Calculate the pseudo labels of the remaining unknown semantic classes, where is the difficult sigmoid function with a linear window of m.

[0101] S50: Train a preset hash model based on the restored training sample set to obtain a cross-modal hash model.

[0102] Specifically, the preset hash model is used to generate a hash code of the modal data, and the cross-modal hash model is trained by recovering the training sample set, wherein the training process of the preset hash model can adopt the training process of the existing cross-modal hash model.

[0103] In the embodiment of the present application, in order to avoid the problem that there may be sample pairs whose similarity cannot be determined due to the existence of unknown labels, the pairwise similarity can be calibrated by expanding the positive and negative relationships. Figure 2 As shown, the training of the preset hash model based on the restored training sample set to obtain the cross-modal hash model specifically includes:

[0104] For each training batch in the restored training sample set, obtain the modal features of each single modal data in the training samples in the training batch;

[0105] According to the restored labels of each training sample in the training batch, the complementary scores between each two training samples are calculated to obtain the complementary score matrix;

[0106] For each training sample in the training batch, determining a complementary training sample corresponding to the training sample according to the complementary score matrix, fusing the modal feature of the training sample with the model features of all complementary training samples to obtain an updated modal feature, and fusing the restored training sample with the restored all complementary training samples to obtain an updated label;

[0107] Based on the updated modal features and updated labels of each training sample, a loss function term is constructed, and a preset hash model is trained based on the loss function term to obtain a cross-modal hash model.

[0108] Specifically, the complementary score matrix includes the complementary scores between each two training samples, wherein the complementary scores can be determined by the asymmetry of the restored labels, and its expression can be:

[0109]

[0110] in, Used to perform 0-1 masking, setting unknown elements to 1 and other elements to 0, l i represents the restored label of the i-th training sample, l j The recovered label of the j-th training sample, Indicates that only positive semantic labels are set to 1.

[0111] After obtaining the complementary score matrix, the complementary training sample corresponding to each training sample can be selected based on the complementary score matrix, for example, by selecting based on the percentile position on the row of the complementary score matrix, or by selecting a preset number of candidate complementary training samples in descending order from all complementary scores corresponding to the i-th training sample, and then randomly selecting a number of complementary training samples from the candidate complementary training samples. After obtaining the complementary training samples, the modal features and restored labels of the training samples are compensated using the modal features and restored labels corresponding to the complementary training samples to obtain updated labels and updated modal features, where the updated labels and updated modal features can be expressed as:

[0112]

[0113] in, represents the updated modal features of the i-th training sample, represents the modal features of the i-th training sample; represents the modal features corresponding to the complementary training samples, represents the updated label of the i-th training sample, represents the restored label of the i-th training sample; represents the restored label corresponding to the complementary training sample, λ v,t =λ v (λ v,t =λ t ) represents the learnable coefficient of the image (text) modality. Of course, it should be noted that when the training sample includes multiple single-modal data, λ v,t It can also represent the learnable coefficients of other single data modalities. Here we take the training samples including image modality and text modality as an example to illustrate, and and In order to distinguish the labels given by training samples and complementary training samples, Indicates the above l i Similarly, and h i Same.

[0114] Furthermore, when constructing the loss function term, the similarity matrix can be calculated directly based on the updated labels corresponding to each training sample, or the real-valued similarity can be constructed using unknown pairs through semantic propagation based on the updated labels corresponding to each training sample, and then the similarity matrix can be calculated based on s ij =0 and s ij The pseudo negative samples introduced by the ratio >0 may be mixed with the original negative pairs and inhibit model fitting. In the embodiment of the present application, the similarity matrix used in constructing the loss function term is:

[0115]

[0116]

[0117] Among them, S∈([0,1]∪{u}) B*B represents the initial similarity matrix, Indicates s ij =0 and s ij >0, B represents the number of samples in the training batch, represents the updated label of the i-th training sample, The updated label of the jth training sample, C represents the semantic label set corresponding to the restored training sample set, Used to perform 0-1 masking, setting unknown elements to 1 and other elements to 0, R∈{0,1} B×B represents a random mask, represents the Hadamard product.

[0118] In summary, this embodiment provides a training method for a cross-modal hash model that deals with partially missing labels. The training method constructs a learnable preset label prompt, builds a label prompt vector using the preset label prompt and the incomplete labels of the cross-modal training samples, then learns the preset label prompt using the completeness score of the label prompt vector. Finally, based on the learned target label prompt, the incomplete labels of the cross-modal training samples are restored to effectively perceive the missing labels, reduce the impact of incomplete labels on the cross-modal hash model, and improve the model performance of the cross-modal hash model. At the same time, the complementary semantic propagation and adaptive negative pairing strategies are used to reduce unknown pairs to balance the positive-negative pair distribution, thereby further improving the model performance of the cross-modal hash model.

[0119] An embodiment of the present application provides a cross-modal hash retrieval method for partially missing labels, using a cross-modal hash model trained using the above-described training method for a cross-modal hash model for partially missing labels. The cross-modal hash retrieval method for partially missing labels specifically includes:

[0120] receiving unimodal data to be retrieved, and determining a hash code of the unimodal data to be retrieved using the cross-modal hash model;

[0121] According to the similarity between the hash code and each modal hash code in the modal hash code library corresponding to the preset modal database, a preset number of candidate modal data corresponding to the single modal data to be retrieved are selected from the preset modal database.

[0122] Based on the above-mentioned training method for a cross-modal hash model with partially missing labels, this embodiment provides a training device for a cross-modal hash model with partially missing labels, such as Figure 4 As shown, the training device for the cross-modal hashing model with partially missing labels specifically includes:

[0123] A generating module 100 is configured to randomly select an initial label set from the positive semantic labels in the incomplete labels of each training sample in a training sample set, generate a plurality of reference label sets based on the initial label set and the incomplete labels, and construct a label prompt vector corresponding to the initial label set and a reference label prompt vector corresponding to each reference label set according to a preset label prompt, wherein the reference label set includes the positive semantic labels in the initial label set, or includes the positive semantic labels in the initial label set and the negative semantic labels in the incomplete labels;

[0124] A calculation module 200 is used to calculate a first completeness score of the label hint vector of each training sample and a second completeness score of each control label hint vector;

[0125] A learning module 300 is configured to construct a total contrast loss term based on all first completeness scores and all second completeness scores, and learn the preset label prompt based on the total contrast loss term to obtain a target label prompt;

[0126] A recovery module 400 is configured to recover the unknown semantic classes in the incomplete labels of each training sample using the semantic label set corresponding to the training sample set based on the target label prompt to obtain a recovered label of each training sample, thereby obtaining a recovered training sample set;

[0127] The training module 500 is used to train a preset hash model based on the restored training sample set to obtain a cross-modal hash model.

[0128] Based on the above-mentioned training method for a cross-modal hash model to cope with partial label missing, this embodiment provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the training method for a cross-modal hash model to cope with partial label missing as described in the above-mentioned embodiment.

[0129] Based on the above-mentioned training method of the cross-modal hash model for partially missing labels, the present application also provides a terminal device, such as Figure 5 As shown, it includes at least one processor 20; a display screen 21; and a memory 22. It may also include a communications interface 23 and a bus 24. The processor 20, display screen 21, memory 22, and communications interface 23 can communicate with each other via bus 24. The display screen 21 is configured to display a preset user guidance interface in the initial setup mode. The communications interface 23 can transmit information. The processor 20 can call the logic instructions in the memory 22 to execute the method in the above embodiment.

[0130] In addition, the logic instructions in the memory 22 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0131] The memory 22, as a computer-readable storage medium, can be configured to store software programs or computer-executable programs, such as program instructions or modules corresponding to the methods in the embodiments of the present disclosure. The processor 20 executes the software programs, instructions, or modules stored in the memory 22 to perform functional applications and data processing, thereby implementing the methods in the above embodiments.

[0132] The memory 22 may include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function; the data storage area may store data created based on the use of the terminal device. In addition, the memory 22 may include high-speed random access memory and non-volatile memory. For example, various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, may also be transient storage media.

[0133] In addition, the specific process of loading and executing the multiple instructions in the storage medium and the processor in the terminal device has been described in detail in the above method and will not be described here one by one.

[0134] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A training method for a cross-modal hashing model with partially missing labels, characterized in that: The training method for the cross-modal hashing model with partially missing labels specifically includes: For each training sample in the training sample set, an initial label set is randomly selected from the positive semantic labels in the incomplete labels of the training sample, a plurality of reference label sets are generated based on the initial label set and the incomplete labels, and a label prompt vector corresponding to the initial label set and a reference label prompt vector corresponding to each reference label set are constructed according to a preset label prompt, wherein the reference label set includes the positive semantic labels in the initial label set, or includes the positive semantic labels in the initial label set and the negative semantic labels in the incomplete labels; Calculating the first completeness score of the label hint vector of each training sample and the second completeness score of each control label hint vector; constructing a total contrast loss term based on all first complete scores and all second complete scores, and learning the preset label prompt based on the total contrast loss term to obtain a target label prompt; Based on the target label prompt, the unknown semantic classes in the incomplete labels of each training sample are recovered by using the semantic label set corresponding to the training sample set to obtain the recovered labels of each training sample, so as to obtain the recovered training sample set; Training a preset hash model based on the restored training sample set to obtain a cross-modal hash model, wherein the cross-modal hash model is used to process image and text data; The step of generating a plurality of comparison label sets based on the initial label set and the incomplete label specifically includes: Performing at least one adjustment operation on the initial label set to obtain a control label set, wherein the adjustment operation is a random addition of negative semantic labels and / or a random deletion of positive semantic labels; Repeating at least one adjustment operation on the initial label set until a plurality of control label sets are obtained.

2. The training method for a cross-modal hashing model for partially missing labels according to claim 1 is characterized in that: The preset label prompt is: Among them, P c represents the learnable prompt for the cth semantic label, P head represents an agnostic prefix of a learnable semantic label, P tail Indicates the agnostic suffix of the learnable semantic label, C desc represents the description label set in the preset label prompt, σ(·) represents the permutation on the description label set, All represent learnable parameters, m, n, m a and n a All are positive integers.

3. The training method for a cross-modal hashing model for partially missing labels according to claim 1, characterized in that: The first completeness score and the second completeness score are both obtained based on a completeness score calculation formula, where the completeness score calculation formula is: Φ(K)=E t (P(K)) Τ h / t Among them, K represents the label set, E t (·) represents the text encoder, P(K) represents the label prompt vector generated based on the label set K, τ represents the temperature parameter, and h represents the modal feature corresponding to the training sample.

4. The training method for a cross-modal hashing model for partially missing labels according to claim 1, characterized in that: The step of recovering the unknown semantic classes in the incomplete labels of each training sample based on the target label prompt and using the semantic label set corresponding to the training sample set to obtain the recovered labels of each training sample, so as to obtain the recovered training sample set, specifically includes: For each training sample in the training sample set, determining all positive semantic labels and all unknown semantic labels corresponding to the training sample based on the semantic label set of the training sample set and the incomplete label of the training sample; With the goal of achieving the highest completeness score, the restored labels of the unknown semantic classes in the incomplete labels of the training samples are searched in the semantic label set consisting of all positive semantic labels and all unknown semantic labels to obtain the restored labels corresponding to the training samples; The restored label of each training sample is used to replace its corresponding incomplete label to obtain a restored training sample set.

5. The training method for a cross-modal hashing model for partially missing labels according to claim 1, wherein: The training of the preset hash model based on the restored training sample set to obtain the cross-modal hash model specifically includes: For each training batch in the restored training sample set, obtain the modal features of each single modal data in the training samples in the training batch; According to the restored labels of each training sample in the training batch, the complementary scores between each two training samples are calculated to obtain the complementary score matrix; For each training sample in the training batch, determining a complementary training sample corresponding to the training sample according to the complementary score matrix, fusing the modal feature of the training sample with the model features of all complementary training samples to obtain an updated modal feature, and fusing the restored training sample with the restored all complementary training samples to obtain an updated label; Based on the updated modal features and updated labels of each training sample, a loss function term is constructed, and a preset hash model is trained based on the loss function term to obtain a cross-modal hash model.

6. The method for training a cross-modal hashing model for partially missing labels according to claim 5, wherein: The similarity matrix used in constructing the loss function term is: Among them, S∈([0,1]Y{u}) B*B represents the initial similarity matrix, r represents s ij =0 and s ij >0, B represents the number of samples in the training batch, represents the updated label of the i-th training sample, The updated label of the jth training sample, C represents the semantic label set corresponding to the restored training sample set, Indicates the execution of a 0-1 mask, setting unknown elements to 1 and other elements to 0, R∈{0,1} B×B represents a random mask, represents the Hadamard product, s ij Represents real-valued similarity.

7. A cross-modal hash retrieval method for partially missing labels, characterized in that: Using a cross-modal hash model trained using the training method for a cross-modal hash model for dealing with partial label loss according to any one of claims 1 to 6, the cross-modal hash retrieval method for dealing with partial label loss specifically includes: receiving unimodal data to be retrieved, and determining a hash code of the unimodal data to be retrieved using the cross-modal hash model; According to the similarity between the hash code and each modal hash code in the modal hash code library corresponding to the preset modal database, a preset number of candidate modal data corresponding to the single modal data to be retrieved are selected from the preset modal database.

8. A training device for a cross-modal hashing model with partially missing labels, characterized in that: The training device for the cross-modal hashing model with partially missing labels specifically includes: a generation module configured to randomly select, for each training sample in a training sample set, an initial label set from the positive semantic labels in the incomplete labels of the training sample, generate a plurality of reference label sets based on the initial label set and the incomplete labels, and construct a label prompt vector corresponding to the initial label set and a reference label prompt vector corresponding to each reference label set according to a preset label prompt, wherein the reference label set includes the positive semantic labels in the initial label set, or includes the positive semantic labels in the initial label set and the negative semantic labels in the incomplete labels; a calculation module, configured to calculate a first completeness score of the label hint vector of each training sample and a second completeness score of each control label hint vector; a learning module, configured to construct a total contrast loss term based on all first completeness scores and all second completeness scores, and learn the preset label prompt based on the total contrast loss term to obtain a target label prompt; a recovery module, configured to recover the incomplete label of each training sample based on the target label prompt and the semantic label set corresponding to the training sample set to obtain a recovered label of each training sample, and replace the incomplete label of each training sample with the recovered label of each training sample to obtain a recovered training sample set; A training module, configured to train a preset hash model based on the restored training sample set to obtain a cross-modal hash model, wherein the cross-modal hash model is used to process image and text data; The step of generating a plurality of comparison label sets based on the initial label set and the incomplete label specifically includes: Performing at least one adjustment operation on the initial label set to obtain a control label set, wherein the adjustment operation is a random addition of negative semantic labels and / or a random deletion of positive semantic labels; Repeating at least one adjustment operation on the initial label set until a plurality of control label sets are obtained.

9. A terminal device, characterized in that: include: processor and memory; The memory stores a computer-readable program executable by the processor; When the processor executes the computer-readable program, it implements the steps of the training method of the cross-modal hash model for dealing with partial missing labels as described in any one of claims 1-6, and / or implements the steps of the cross-modal hash retrieval method for dealing with partial missing labels as described in claim 7.

Images