Dynamic cross-modal hash retrieval method and system based on semi-paired data

Abstract

The invention discloses a dynamic cross-modal Hash retrieval method and system based on semi-paired data. The method comprises the following steps: learning a current paired Hash code based on paired label information to obtain a target paired Hash code; learning the current non-paired hash code based on the intra-modal similarity matrix between the paired data set and the non-paired data set and the inter-modal similarity matrix of the non-paired data set to obtain a target non-paired hash code; obtaining a total target Hash code based on the target pairing Hash code and the target non-pairing Hash code; learning a current multi-modal hash function of the multi-modal data set based on the total target hash code and the multi-modal data feature vector to obtain a target multi-modal hash function; and retrieving the to-be-retrieved sample based on the target multi-modal hash function and the total target hash code of the database to obtain a target retrieval result. According to the method, the accuracy and efficiency of cross-modal hash retrieval can be improved, and the method can be widely applied to the technical field of information processing.

Description

Technical Field

[0001] The present application relates to the field of information processing technology, and in particular to a dynamic cross-modal hash retrieval method and system based on semi-paired data. Background Art

[0002] In the related technology, in the field of cross-modal hash retrieval research, there are mainly the following problems:

[0003] (1) The cross-modal hash retrieval method assumes that the data environment is static. However, the actual data environment is complex and changeable, and new data will be continuously added. For the static cross-modal method, if the original hash function remains unchanged, it will not be able to reflect the dynamic changes of the new data compared to the old data, and it will not be able to adapt to the changes in the new data environment, resulting in a decrease in retrieval performance. In addition, the static cross-modal hash method often requires retraining the entire hash model to adapt to the new data environment, but this is undoubtedly very time-consuming, especially when the data environment changes very frequently. This approach of retraining the entire model is inefficient.

[0004] (2) Cross-modal hash retrieval methods assume that the training data is one-to-one corresponding. However, in the actual data environment, it is impossible to guarantee that the multi-modal data is one-to-one corresponding, that is, unpaired. Since most cross-modal hashing methods can only be trained when the multi-modal training data is one-to-one corresponding, they cannot cope with unpaired data environments.

[0005] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0006] The embodiments of the present application aim to solve at least one of the technical problems in the related art to a certain extent. To this end, the main purpose of the embodiments of the present application is to propose a dynamic cross-modal hash retrieval method and system based on semi-paired data, which can improve the accuracy and efficiency of cross-modal hash retrieval.

[0007] To achieve the above purpose, an embodiment of the present application proposes a dynamic cross-modal hash retrieval method based on semi-paired data, and the method includes the following steps:

[0008] Acquire a multimodal dataset; the multimodal dataset includes a paired dataset and an unpaired dataset;

[0009] Performing feature extraction on the multimodal data set to obtain a multimodal data feature vector; the multimodal data feature vector includes a paired feature vector and an unpaired feature vector;

[0010] Learning the current pairing hash code of the pairing data set based on the pairing label information of the pairing data set to obtain a target pairing hash code;

[0011] Based on the intra-modality similarity matrix between the paired data set and the unpaired data set and the inter-modality similarity matrix of the unpaired data set, learning the current unpaired hash code of the unpaired data set to obtain a target unpaired hash code;

[0012] Based on the overall objective function, the target paired hash code and the target unpaired hash code are integrated to obtain an overall target hash code;

[0013] Learning the current multimodal hash function of the multimodal data set based on the total target hash code and the multimodal data feature vector to obtain a target multimodal hash function, and storing the target multimodal hash function and the total target hash code in a database;

[0014] The samples to be retrieved are retrieved based on the target multimodal hash function and the total target hash code in the database to obtain a target retrieval result.

[0015] In some embodiments, extracting features from the multimodal data set to obtain a multimodal data feature vector includes:

[0016] Performing a first feature extraction process on the paired data set in the multimodal data set to obtain the paired feature vector;

[0017] A second feature extraction process is performed on the unpaired data set in the multimodal data set to obtain the unpaired feature vector.

[0018] In some embodiments, the learning of the current pairing hash code of the pairing data set based on the pairing tag information of the pairing data set to obtain the target pairing hash code includes:

[0019] Constructing a pairing label matrix of the pairing data set based on the pairing label information of the pairing data set;

[0020] Normalizing the paired label matrix to obtain a normalized paired label matrix;

[0021] Calculating the pairwise semantic similarity between the label vectors in the normalized paired label matrix to obtain the pairwise semantic similarity value;

[0022] Constructing a pairwise similarity matrix based on the pairwise semantic similarity values;

[0023] The current pairing hash code of the pairing data set is learned based on the pairing similarity matrix to obtain the target pairing hash code.

[0024] In some embodiments, the integrating the target paired hash code and the target unpaired hash code based on the overall objective function to obtain the overall target hash code includes:

[0025] Merge the target paired hash code and the target unpaired hash code to obtain a target merged hash code;

[0026] The target merged hash code is optimized based on the overall objective function to obtain the overall target hash code.

[0027] In some embodiments, the learning of the current multimodal hash function of the multimodal data set based on the total target hash code and the multimodal data feature vector to obtain a target multimodal hash function, and storing the target multimodal hash function and the total target hash code in a database includes:

[0028] Based on the total target hash code and the multimodal data feature vector, the current multimodal hash function of the multimodal data set is learned using a linear regression method to obtain the target multimodal hash function, and the target multimodal hash function and the total target hash code are stored in the database; wherein the target multimodal hash function is used to characterize the target hash function corresponding to various modal data in the multimodal data set.

[0029] In some embodiments, the retrieving the sample to be retrieved based on the target multimodal hash function and the total target hash code in the database to obtain a target retrieval result includes:

[0030] Converting the to-be-retrieved feature vector of the to-be-retrieved sample into a to-be-retrieved hash code based on the target multimodal hash function in the database;

[0031] Calculating the Hamming distance between the to-be-retrieved hash code and the total target hash code in the database to obtain a similarity calculation result;

[0032] The candidate samples in the database are retrieved according to the similarity calculation result to obtain the target retrieval result.

[0033] In some embodiments, searching the candidate samples in the database according to the similarity calculation result to obtain the target search result includes:

[0034] Sorting the candidate samples in the database by similarity according to the similarity calculation result to obtain a similarity ranking result;

[0035] The candidate samples in the similarity ranking results that meet the target sorting rule and the target quantity are used as the target retrieval results.

[0036] To achieve the above purpose, another aspect of the embodiment of the present application proposes a dynamic cross-modal hash retrieval system based on semi-paired data, and the system includes the following modules:

[0037] A data acquisition module, used to acquire a multimodal data set; the multimodal data set includes a paired data set and an unpaired data set;

[0038] A feature extraction module, used to extract features from the multimodal data set to obtain a multimodal data feature vector; the multimodal data feature vector includes a paired feature vector and an unpaired feature vector;

[0039] A pairing hash code learning module, used to learn the current pairing hash code of the pairing data set based on the pairing label information of the pairing data set to obtain a target pairing hash code;

[0040] an unpaired hash code learning module, configured to learn a current unpaired hash code of the unpaired data set based on an intra-modality similarity matrix between the paired data set and the unpaired data set and an inter-modality similarity matrix of the unpaired data set to obtain a target unpaired hash code;

[0041] A total target hash code acquisition module is used to integrate the target paired hash code and the target unpaired hash code based on the total target function to obtain a total target hash code;

[0042] A hash function learning module, configured to learn a current multimodal hash function of the multimodal data set based on the total target hash code and the multimodal data feature vector to obtain a target multimodal hash function, and store the target multimodal hash function and the total target hash code in a database;

[0043] The sample retrieval module is used to retrieve the sample to be retrieved based on the target multimodal hash function and the total target hash code in the database to obtain a target retrieval result.

[0044] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0045] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0046] The embodiments of the present application include at least the following beneficial effects: the present application provides a dynamic cross-modal hash retrieval method and system based on semi-paired data, the scheme is through obtaining a multimodal data set; the multimodal data set includes a paired data set and an unpaired data set; feature extraction is performed on the multimodal data set to obtain a multimodal data feature vector; the multimodal data feature vector includes a paired feature vector and an unpaired feature vector; based on the paired label information of the paired data set, the current paired hash code of the paired data set is learned to obtain a target paired hash code; based on the intra-modal similarity matrix between the paired data set and the unpaired data set and the unpaired data set The inter-modal similarity matrix is ​​used to learn the current unpaired hash code of the unpaired data set to obtain the target unpaired hash code; the target paired hash code and the target unpaired hash code are integrated based on the total target function to obtain the total target hash code; based on the total target hash code and the multimodal data feature vector, the current multimodal hash function of the multimodal data set is learned to obtain the target multimodal hash function, and the target multimodal hash function and the total target hash code are stored in the database; the samples to be retrieved are retrieved based on the target multimodal hash function and the total target hash code in the database to obtain the target retrieval results. The embodiment of the present application obtains the target paired hash code by learning the current paired hash code using the paired label information of the paired data set, thereby maintaining the similarity and semantic consistency of the paired data set in the hash space, and providing a reliable benchmark for subsequent cross-modal retrieval; the target unpaired hash code is obtained by learning the intra-modal similarity matrix between the paired data set and the unpaired data set and the inter-modal similarity matrix of the unpaired data set, which not only utilizes the intra-modal similarity of the paired data and the unpaired data, but also solves the unpaired data problem by utilizing the semantic information between the paired data and the unpaired data of different modalities, fully utilizing the information in the unpaired data, and compensating for the problem of insufficient paired information by constructing a similarity matrix, thereby improving the generalization ability and robustness of the hash code; based on the overall goal The function integrates the target hash codes of paired data and unpaired data to obtain the total target hash code, which combines the information of paired data and unpaired data so that the final hash code can simultaneously reflect the characteristics and relationships of the two types of data, thereby improving the accuracy and efficiency of cross-modal retrieval; based on the total target hash code and the multimodal data feature vector, the current multimodal hash function of the multimodal dataset is learned to obtain the target multimodal hash function, so that the hash function can be applied to data of multiple modalities and can capture the semantic relationship between different modalities, providing data support for cross-modal retrieval; finally, the samples to be retrieved can be dynamically retrieved based on the continuously updated target multimodal hash function and the total target hash code in the database, thereby achieving fast, accurate and real-time cross-modal retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a flowchart of the steps of a dynamic cross-modal hash retrieval method based on semi-paired data provided in an embodiment of the present application;

[0048] Figure 2 It is a schematic diagram of the process of hash coding learning based on semi-paired data provided in an embodiment of the present application;

[0049] Figure 3 It is a structural schematic diagram of a dynamic cross-modal hash retrieval system based on semi-paired data provided in an embodiment of the present application;

[0050] Figure 4 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of systems and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0052] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0053] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0055] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0056] (1) Cross-modal retrieval: a retrieval method that uses samples from one modality to retrieve samples from other modalities.

[0057] (2) Hash retrieval: The hash algorithm maps the samples in the data set from the initial high-dimensional feature space to the low-dimensional binary Hamming space, generates a set of compact hash codes for each sample, and evaluates the similarity between samples based on the Hamming distance between the sample hash codes, thereby achieving fast nearest neighbor retrieval of the query sample. The core of the hash algorithm is to train a set of hash functions, each hash function corresponds to a hash hyperplane, and divides the initial feature space into two parts, one side corresponds to hash code 1, and the other side corresponds to hash code 0. Therefore, each hash function corresponds to a hash code.

[0058] (3) Dynamic data environment: The data set or data environment used for model training changes over time. For example, as time goes by, new images and texts are constantly added to the original data set, and old images and texts may be constantly deleted from the data set. Therefore, the dynamic data environment is more complex than the static data environment.

[0059] (4) Semi-paired: The cross-modal hash retrieval model trains the model on multimodal data during the training phase to obtain a hash function. If each modality of these multimodal data has a one-to-one correspondence and no missing data, it can be regarded as paired data. If some modal data in the multimodal data are missing, it is called semi-paired data.

[0060] (5) SPI H (Semi-paired Hashing for Cross-modal Retrieval in Non-statistical Environment), the SPI H method is a cross-modal dynamic hash retrieval method proposed in an embodiment of the present application.

[0061] In the related art, in the field of cross-modal hash retrieval research, the following problems exist: (1) The cross-modal hash retrieval method assumes that the data environment is static, but the actual data environment is complex and changeable, and new data will be continuously added. For static cross-modal methods, if the original hash function remains unchanged, it cannot reflect the dynamic changes of new data compared with old data, and cannot adapt to changes in the new data environment, resulting in a decrease in retrieval performance. In addition, static cross-modal hash methods often require retraining the entire hash model to adapt to the new data environment, but this is undoubtedly very time-consuming, especially when the data environment changes very frequently. This approach of retraining the entire model is inefficient. (2) The cross-modal hash retrieval method assumes that the training data is one-to-one corresponding, but in the actual data environment, it cannot be guaranteed that the multi-modal data is one-to-one corresponding, that is, unpaired. Since most cross-modal hash methods can only be trained when the multi-modal training data is one-to-one corresponding, they cannot cope with unpaired data environments.

[0062] In view of this, an embodiment of the present application provides a dynamic cross-modal hash retrieval method and system based on semi-paired data, the scheme is through obtaining a multimodal data set; the multimodal data set includes a paired data set and an unpaired data set; feature extraction is performed on the multimodal data set to obtain a multimodal data feature vector; the multimodal data feature vector includes a paired feature vector and an unpaired feature vector; based on the paired label information of the paired data set, the current paired hash code of the paired data set is learned to obtain a target paired hash code; based on the intra-modal similarity matrix between the paired data set and the unpaired data set and the inter-modal similarity matrix of the unpaired data set, the current unpaired hash code of the unpaired data set is learned to obtain a target unpaired hash code; based on the total target function, the target paired hash code and the target unpaired hash code are integrated to obtain a total target hash code; based on the total target hash code and the multimodal data feature vector, the current multimodal hash function of the multimodal data set is learned to obtain a target multimodal hash function, and the target multimodal hash function and the total target hash code are stored in a database; based on the target multimodal hash function and the total target hash code in the database, a sample to be retrieved is retrieved to obtain a target retrieval result. The embodiment of the present application obtains the target paired hash code by learning the current paired hash code using the paired label information of the paired data set, thereby maintaining the similarity and semantic consistency of the paired data set in the hash space, and providing a reliable benchmark for subsequent cross-modal retrieval; the target unpaired hash code is obtained by learning the intra-modal similarity matrix between the paired data set and the unpaired data set and the inter-modal similarity matrix of the unpaired data set, which not only utilizes the intra-modal similarity of the paired data and the unpaired data, but also solves the unpaired data problem by utilizing the semantic information between the paired data and the unpaired data of different modalities, fully utilizing the information in the unpaired data, and compensating for the problem of insufficient paired information by constructing a similarity matrix, thereby improving the generalization ability and robustness of the hash code; based on the overall goal The function integrates the target hash codes of paired data and unpaired data to obtain the total target hash code, which combines the information of paired data and unpaired data so that the final hash code can simultaneously reflect the characteristics and relationships of the two types of data, thereby improving the accuracy and efficiency of cross-modal retrieval; based on the total target hash code and the multimodal data feature vector, the current multimodal hash function of the multimodal dataset is learned to obtain the target multimodal hash function, so that the hash function can be applied to data of multiple modalities and can capture the semantic relationship between different modalities, providing data support for cross-modal retrieval; finally, the samples to be retrieved can be dynamically retrieved based on the continuously updated target multimodal hash function and the total target hash code in the database, thereby achieving fast, accurate and real-time cross-modal retrieval.

[0063] The dynamic cross-modal hash retrieval method based on semi-paired data provided in the embodiment of the present application relates to the field of information processing technology. The dynamic cross-modal hash retrieval method based on semi-paired data provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content De l ivery Network, content distribution network) and cloud servers for basic cloud computing services such as big data and artificial intelligence platforms, and the server can also be a node server in a blockchain network; the software can be an application that implements a dynamic cross-modal hash retrieval method based on semi-paired data, etc., but is not limited to the above form.

[0064] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs (personal computers), minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0065] See also Figure 1 , Figure 1 is an optional step flow chart of a dynamic cross-modal hash retrieval method based on semi-paired data provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S107.

[0066] Step S101, obtaining a multimodal dataset; the multimodal dataset includes a paired dataset and an unpaired dataset;

[0067] Among them, a multimodal dataset refers to a data set containing multiple different modalities (or types), and the modalities can be text, image, audio, video, etc. In the embodiment of the present application, the multimodal dataset includes a paired dataset and an unpaired dataset, wherein the paired dataset contains paired data points, and the paired data points are related or paired in a certain sense. For example, a certain text data can be paired one-to-one with a certain image data; there is no clear pairing relationship between the data points contained in the unpaired dataset, and these data points may come from different sources, different time periods or different objects. For example, a certain text data cannot be paired with a certain image data, and they are all independent data.

[0068] In specific applications, the data set or data environment for model training changes over time. For example, assuming that new images and texts are constantly added to the original data set as time goes by, old images and texts may also be constantly deleted from the data set. In this way, the dynamic data environment is more complex than the static data environment. In the embodiment of the present application, SPIC (Semi-paired Hash ing for Cross-modal Retrieval Non-static Environment) is proposed, that is, the dynamic cross-modal hash retrieval method based on semi-paired data mentioned in the embodiment of the present application. For the dynamic data environment, the retrieval model can adapt to the new data environment, and the hash function and hash code can be updated according to the changes in the data environment, thereby improving the accuracy and efficiency of the retrieval. This is the main difference between the dynamic hash retrieval method and the static hash retrieval method.

[0069] Step S101 shown in the embodiment of the present application provides a diversified data source for subsequent cross-modal retrieval by acquiring a multimodal dataset including a paired dataset and an unpaired dataset.

[0070] Step S102, performing feature extraction on the multimodal data set to obtain a multimodal data feature vector; the multimodal data feature vector includes a paired feature vector and an unpaired feature vector;

[0071] In some embodiments, step S102 may include: performing a first feature extraction process on a paired data set in the multimodal data set to obtain a paired feature vector; performing a second feature extraction process on an unpaired data set in the multimodal data set to obtain an unpaired feature vector.

[0072] The first feature extraction process refers to the feature extraction operation performed on the paired data set, which aims to extract feature information useful for subsequent tasks (such as hash coding learning, retrieval, etc.) from the original paired data set, namely, the paired feature vector. The second feature extraction process is the feature extraction operation performed on the unpaired data set to obtain the unpaired feature vector. Although there is no direct pairing relationship within the unpaired data set, the extracted features can reflect the intrinsic properties and differences of the data.

[0073] Step S102 shown in the embodiment of the present application performs feature extraction on paired data sets and unpaired data sets to obtain corresponding feature vectors, thereby providing effective data representation for subsequent hash coding learning, while being able to remove redundant information, retain key features, and improve the accuracy and efficiency of hash coding learning.

[0074] Step S103, learning the current pairing hash code of the pairing data set based on the pairing tag information of the pairing data set to obtain a target pairing hash code;

[0075] In some embodiments, step S103 may include: constructing a pairing label matrix of the pairing data set based on the pairing label information of the pairing data set; normalizing the pairing label matrix to obtain a normalized pairing label matrix; calculating the pairwise semantic similarity between the label vectors in the normalized pairing label matrix to obtain a pairwise semantic similarity value; constructing a pairwise similarity matrix based on the pairwise semantic similarity value; and learning the current pairing hash code of the pairing data set based on the pairwise similarity matrix to obtain a target pairing hash code.

[0076] The pairing label information is the label information of the paired data set, and the pairing label information can indicate whether two different data viewpoints belong to the same category, are similar, or match each other. The pairing label matrix is ​​a two-dimensional matrix used to represent the pairing label information between all data points in the paired data set.

[0077] Among them, pairwise semantic similarity refers to the similarity between two data points at the semantic level. In hash coding or retrieval tasks, pairwise semantic similarity is usually evaluated based on paired label information. The pairwise semantic similarity value is a quantitative indicator used to represent the pairwise semantic similarity between two data points. The pairwise similarity matrix is ​​a two-dimensional matrix used to represent the pairwise semantic similarity values ​​between all data points in the paired dataset, similar to the paired label matrix.

[0078] Among them, the current paired hash code refers to the current hash code of the paired data set during the learning process. These codes may not be optimal or updated in real time, and further learning is required to optimize or iterate to obtain the target paired hash code. The target paired hash code refers to the optimal hash code finally obtained through the learning process. The optimal hash code can effectively represent the data points in the paired data set and show good performance in hash coding or retrieval tasks. The target paired hash code is learned through an optimization algorithm based on the paired label information.

[0079] It should be noted that in a dynamic data environment, the target pairing hash code is dynamically updated.

[0080] In the specific implementation, the hash coding learning process of paired data is as follows:

[0081] In order to learn the hash coding of paired data, it is first necessary to obtain the label information of the paired data and construct the pairwise similarity matrix of the paired data, and give the corresponding objective function, where the objective function is a loss function. By solving the objective function, the final result of the corresponding matrix can be obtained, that is, by minimizing the objective function, the hash coding corresponding to the paired data of different modalities can be obtained. In the SPI H method, the pairwise similarity matrix of paired data must first be constructed to guide the learning of hash coding.

[0082] Specifically, the paired label information of the paired data set is used to construct a paired label matrix, and the paired label matrix is ​​used to construct a pairwise similarity matrix. The hash code corresponding to the paired data set can be learned using the pairwise similarity matrix and the paired label matrix.

[0083] Among them, two samples sharing similar labels are expected to have similar hash codes and a small Hamming distance. Therefore, the cosine similarity between label vectors is used to evaluate the pairwise semantic similarity between two samples. The specific calculation is as follows:

[0084]

[0085] in, Represents the pairwise similarity matrix of paired data; represents the 2-norm normalized label matrix with a time step of t, where 2-norm represents the second norm (Frobenius norm); 1 represents an all-one matrix; represents the transpose of an all-ones matrix multiplied by another all-ones matrix; Indicates transpose.

[0086] Among them, the calculation method of the 2-norm normalized label matrix is ​​as follows:

[0087]

[0088] in, express The i-th column vector of (the total label matrix corresponding to the paired data at time t); express The i-th column vector of the 2-norm normalized label matrix with time step t.

[0089] Specifically, the similarity between paired data can be maintained by minimizing the difference between hash code similarity and pairwise semantic similarity, which is calculated as follows:

[0090]

[0091] Among them, B1 (t) represents the total pairing matrix of the hash code matrix corresponding to the paired data at time t, r represents the B matrix, that is, the number of bits of hash code in the hash code matrix (hash code length), Represents the pairwise similarity matrix (also called pairwise semantic similarity matrix) of paired data. Since B1 (t) is a binary matrix. Minimizing the above equation (3) is an NP (Non-deterministic Polynomial time) problem. Therefore, the embodiment of the present application introduces a real-valued matrix V (t) Replacing one of the hash code matrices makes the optimization process more efficient, and the real-valued matrix V (t) The potential representation of the hash code B1 is retained. Therefore, formula (3) can be transformed as follows:

[0092]

[0093] in, It is represented as a real-valued matrix with a value range of -1 to 1; I represents the identity matrix.

[0094] At the same time, in order to minimize B1 (t) and V (t) In order to reduce the information loss between , the embodiment of the present application adds a regularization term of the orthogonal rotation matrix R. The calculation formula for minimizing the information loss is as follows:

[0095]

[0096] Among them, stR (t) R (t)T represents the orthogonalization restriction on the orthogonal rotation matrix R.

[0097] In addition, each paired data has unique corresponding label information, which is conducive to learning hash coding to preserve more accurate semantic information. Therefore, linear embedding is learned through hash coding to preserve label semantic information. The calculation formula is as follows:

[0098]

[0099] Among them, P (t) Represents a linear embedding matrix, which can be learned and updated online through iterative optimization. However, the label matrix is ​​actually a binary matrix, and the learning efficiency of the linear embedding matrix is ​​low. Directly embedding the potential representation into the label will result in high quantization loss. Therefore, the embodiment of the present application introduces an adaptive margin factor to reduce the quantization loss and expand the margins of different categories. The calculation formula of (index matrix) is as follows:

[0100]

[0101] Among them, "+1" indicates the positive direction, and "-1" indicates the negative direction; j indicates the jth row of the total label matrix corresponding to the paired data at time t, and k indicates the kth column of the total label matrix corresponding to the paired data at time t.

[0102] Among them, the marginal matrix represents the marginal size V of each potential representation (t) , Among them, ε jk represents the value of the jth row and kth column of the marginal matrix. Therefore, formula (6) can be transformed into the following formula:

[0103]

[0104] Wherein, ⊙ represents the Hadamard product operator.

[0105] In step S103 shown in the embodiment of the present application, the target pairing hash code is obtained by learning the current pairing hash code using the pairing label information of the pairing data set, thereby maintaining the similarity and semantic consistency of the pairing data set in the hash space, and providing a reliable benchmark for subsequent cross-modal retrieval.

[0106] Step S104, based on the intra-modality similarity matrix between the paired data set and the unpaired data set and the inter-modality similarity matrix of the unpaired data set, learning the current unpaired hash code of the unpaired data set to obtain a target unpaired hash code;

[0107] The intra-modality similarity matrix is ​​a measure of the similarity between data points in the same modality (e.g., all images, all texts, etc.). Specifically, the intra-modality similarity matrix can be a similarity relationship matrix between paired image data and unpaired image data, or a similarity relationship matrix between paired text data and unpaired text data.

[0108] The inter-modal similarity matrix refers to the measure of the similarity of data points between different modalities. An unpaired dataset may contain data from multiple different modalities (such as images and text), and the inter-modal similarity matrix is ​​used to capture the association or similarity of these data points between different modalities. Specifically, the inter-modal similarity matrix can be an inter-modal similarity relationship matrix between unpaired image data and unpaired text data.

[0109] Among them, the current unpaired hash code refers to the current hash code of the unpaired dataset during the learning process. The current unpaired hash code may not be optimal or updated in real time, and further learning is required to optimize or iterate to obtain the target unpaired hash code. The target unpaired hash code refers to the optimal hash code finally obtained through the learning process. The target unpaired hash code can effectively represent the data points in the unpaired dataset and show good performance in tasks such as hash coding or cross-modal retrieval. The target unpaired hash code is learned based on the intra-modal similarity matrix and the inter-modal similarity matrix.

[0110] In the specific implementation, the specific process of hash coding learning for unpaired data is as follows:

[0111] In the SPIC method, in addition to the similarity information between paired data, the similarity information provided by unpaired data is also of great significance for learning discriminative hash codes. Unlike the paired data in which each pair of data shares a unified hash code, unpaired data of different modalities have their own hash codes, such as the unpaired image data in the unpaired data has a corresponding hash code, and the unpaired text data has a corresponding hash code. Therefore, in the SPIC method provided in the embodiment of the present application, the intra-modal similarity matrix between the unpaired data and the paired data of each modality, as well as the inter-modal similarity matrix between the unpaired image data and the unpaired text data, can be provided to hash code learning, and a corresponding objective function is given. By minimizing the objective function, the hash codes corresponding to the unpaired data of different modalities can be obtained.

[0112] Among them, the label information of paired data and the label information of unpaired data (such as unpaired image label information and unpaired text label information) can be used to obtain the intra-modal similarity matrix between unpaired data and paired data and the inter-modal similarity matrix between unpaired image data and unpaired text data.

[0113] Specifically, first, a paired label matrix is ​​constructed based on the paired label information of the paired data set, and the paired label matrix is ​​used to construct a pairwise similarity matrix; then, an unpaired label matrix of unpaired data is constructed based on the label information of the unpaired data (such as unpaired image label information and unpaired text label information); based on the paired label matrix and the unpaired label matrix, an intra-modal similarity matrix between unpaired data and paired data, and an inter-modal similarity matrix between unpaired image data and unpaired text data can be constructed; finally, based on the paired label matrix, the unpaired label matrix, the intra-modal similarity matrix between unpaired data and paired data, and the inter-modal similarity matrix between unpaired image data and unpaired text data, the hash codes corresponding to the unpaired data can be learned.

[0114] For intra-modality similarity, similar to formula (4), the similarity loss minimization formula between unpaired data and paired data of two modalities is as follows:

[0115]

[0116] in, represents the hash code of the unpaired image, represents the hash code of the unpaired text, Represents the similarity relationship matrix (semantic similarity matrix) between paired image data and unpaired image data, Represents the similarity relationship matrix (semantic similarity matrix) between paired text data and unpaired text data.

[0117] In order to further explore the semantic information between different modalities of unpaired data, the embodiment of the present application can also learn the similarity between modalities, as follows:

[0118]

[0119] in, Represents the similarity relationship matrix (semantic similarity matrix) between unpaired image data and unpaired text data. It is worth noting that the optimization formula (10) is not an NP-hard problem and has an efficient discrete optimization solution.

[0120] Step S104 shown in the embodiment of the present application, by introducing the intra-modal similarity matrix between the paired data set and the unpaired data set and the inter-modal similarity matrix of the unpaired data set to learn and obtain the target unpaired hash code, is of great significance for learning discriminative hash codes. This step not only utilizes the intra-modal similarity of paired data and unpaired data, but also solves the unpaired data problem by utilizing the semantic information between paired data and unpaired data of different modalities, highlights the correlation between them, and makes full use of the information in the unpaired data. By constructing a similarity matrix to make up for the problem of insufficient pairing information, the generalization ability and robustness of the hash code are improved.

[0121] Step S105, integrating the target paired hash code and the target unpaired hash code based on the overall objective function to obtain an overall target hash code;

[0122] With respect to the overall objective function, in combination with the above formulas (4), (5), (8), (9) and (10), the overall objective function of the SPI H method of the embodiment of the present application is as follows:

[0123]

[0124] Among them, α1, α2, β1, β2, β3 are all trade-off parameters; assume that a new data block appears at each time step, and the data block consists of two modalities (i.e., image and text), among which some images and texts are paired, and the rest are not paired. Assume that the number of paired image-text data, unpaired images, and unpaired texts in each time step remains unchanged, n1 represents the number of paired image-text data in each time step, n2 represents the number of unpaired image data in each time step, and n3 represents the number of unpaired text data in each time step. By minimizing the above formula (11), the hash codes of paired data and unpaired data can be learned.

[0125] In addition, the fine-grained similarity matrix is ​​constructed by the 2-norm normalized label matrix and can be rewritten as follows:

[0126]

[0127] Therefore, the similarity matrix can be rewritten in block form as follows:

[0128]

[0129] in, is the similarity matrix between old data and old data, is the similarity matrix between old data and new data, is the similarity matrix between new data and old data, is the similarity matrix between new data and new data. Therefore, the overall objective function can be further rewritten as follows:

[0130]

[0131] in, Represents the label matrix corresponding to the paired data, when Expressed as hour, Represents the label matrix corresponding to the unpaired image modality, when Expressed as hour, Represents the label matrix corresponding to the unpaired text, where c represents the total number of label categories. Represents the hash code corresponding to the paired data, Represents the hash code corresponding to the unpaired image data, Represents the hash code corresponding to the unpaired text data.

[0132] In some embodiments, step S105 may include: merging the target paired hash code and the target unpaired hash code to obtain a target merged hash code; and optimizing the target merged hash code based on the overall objective function to obtain an overall target hash code.

[0133] Among them, the target merged hash code refers to a preliminary hash code representation, which is obtained by merging the target paired hash code (i.e., the optimal hash code learned from paired data) and the target unpaired hash code (i.e., the optimal hash code learned from unpaired data).

[0134] Among them, the overall target hash code refers to a hash code that can simultaneously reflect the similarity between paired data and the semantic association between unpaired data (including between different modalities). It aims to improve the discriminability, generalization ability and cross-modal retrieval performance of hash codes by integrating information from different data sources and modalities.

[0135] In a specific implementation, based on minimizing the objective function, hash codes corresponding to paired data and unpaired data of different modalities can be obtained, and then the paired hash codes and unpaired hash codes of the same modality can be combined to obtain hash codes of the same modality. Among them, the overall objective function (total objective function) is the sum of the objective functions involved in step S103 and step S104. The embodiment of the present application can learn hash codes of different modalities by solving the overall objective function.

[0136] In step S105 shown in the embodiment of the present application, the target hash codes of the paired data and the unpaired data are integrated based on the overall objective function to obtain an overall target hash code. This process comprehensively considers the information of the paired data and the unpaired data, so that the hash code finally obtained can simultaneously reflect the characteristics and relationship of the two types of data, thereby improving the accuracy and efficiency of cross-modal retrieval.

[0137] Step S106, learning the current multimodal hash function of the multimodal data set based on the total target hash code and the multimodal data feature vector to obtain a target multimodal hash function, and storing the target multimodal hash function and the total target hash code in a database;

[0138] In some embodiments, step S106 may include: learning the current multimodal hash function of the multimodal data set using a linear regression method based on the total target hash code and the multimodal data feature vector to obtain a target multimodal hash function, and storing the target multimodal hash function and the total target hash code in a database; wherein the target multimodal hash function is used to characterize the target hash function corresponding to various modal data in the multimodal data set.

[0139] Among them, the current multimodal hash function refers to a hash function that already exists or is preliminarily defined before the hash function learning begins. The current multimodal hash function needs to be optimized and learned to obtain the target multimodal hash function. Among them, the current multimodal hash function is the current hash function corresponding to various modal data in the multimodal dataset. The target multimodal hash function is used to characterize the target hash function corresponding to various modal data in the multimodal dataset. The target multimodal hash function refers to the hash function obtained through the learning process, which can better map the various modal data in the multimodal dataset to the same hash space, and maintain the similarity and semantic relevance between the data in this space. The target multimodal hash function aims to improve the discriminability, generalization ability and cross-modal retrieval performance of hash coding.

[0140] Among them, a database is a collection of data storage and management, allowing users to store, retrieve, modify and delete data in a structured manner. In an embodiment of the present application, the database is used to store the learned hash function (such as the target multimodal hash function) and the corresponding hash code (such as the total target hash code) for subsequent data processing, retrieval and analysis tasks.

[0141] In the specific implementation, the specific engineering of hash function learning is as follows:

[0142] After obtaining the hash codes of all new data, we can obtain the hash function corresponding to each mode at the learning time t. For each mode, the hash codes of paired data and unpaired data are combined to learn the hash function. Assume and They represent the hash code of the image mode and the hash code of the text mode at time t respectively, and the two can be combined by the following calculation method:

[0143]

[0144] in, and It can be expressed as the total hash coding matrix of the image modality at time t and the total hash coding matrix of the text modality respectively.

[0145] When the hash codes of each modality are known, the hash function corresponding to the modality can be obtained by minimizing the following formula:

[0146]

[0147] in, represents the hash function corresponding to the mth mode, which is used to map the high-dimensional feature matrix to the low-dimensional Hamming space (hash coding). It can be expressed as the total characteristic data matrix corresponding to the mth mode at time t:

[0148]

[0149] Where m = 1, 2, represents the feature matrix of paired data, represents the unpaired image feature matrix, Represents the unpaired text feature matrix, where d1 represents the feature matrix dimension of the image and d2 represents the feature matrix dimension of the text.

[0150] Among them, at time t, It can represent the total feature matrix of paired data from the beginning to the current time t, which is composed of the total feature matrix before time t and the feature matrix at time t.

[0151] at last, It can be obtained by taking partial derivatives of formulas (15) and (16) and setting them to 0. The specific calculation formula is as follows:

[0152]

[0153] In step S106 shown in the embodiment of the present application, the current multimodal hash function of the multimodal data set is learned based on the total target hash code and the multimodal data feature vector to obtain the target multimodal hash function. This process enables the hash function to be applicable to data of multiple modalities and to capture the semantic relationship between different modalities, thereby providing data support for cross-modal retrieval.

[0154] Step S107, searching the samples to be searched based on the target multimodal hash function and the total target hash code in the database to obtain a target search result.

[0155] In some embodiments, step S107 may include: converting the feature vector to be retrieved of the sample to be retrieved into a hash code to be retrieved based on the target multimodal hash function in the database; calculating the Hamming distance between the hash code to be retrieved and the total target hash code in the database to obtain a similarity calculation result; retrieving candidate samples in the database according to the similarity calculation result to obtain a target retrieval result.

[0156] In some specific embodiments, retrieving candidate samples in a database based on similarity calculation results to obtain target retrieval results may include: sorting the candidate samples in the database by similarity based on the similarity calculation results to obtain similarity ranking results; and taking the candidate samples in the similarity ranking results that meet the target sorting rules and target quantity as the target retrieval results.

[0157] Among them, the feature vector to be retrieved is a feature vector extracted according to the feature extraction method corresponding to the sample to be retrieved. Exemplarily, if the sample to be retrieved is image data, the image feature vector is extracted according to the feature extraction method corresponding to the image data; if the sample to be retrieved is text data, the text feature vector is extracted according to the feature extraction method corresponding to the text data. With respect to the feature extraction method, the embodiment of the present application does not limit this, such as: for images, deep learning models such as convolutional neural networks can be used to extract features; for texts, word embedding or pre-trained text encoders can be used to extract features. Among them, the hash code to be retrieved is the hash code of the sample to be retrieved obtained by a hash function based on the modality corresponding to the sample to be retrieved.

[0158] Among them, the Hamming distance is used to measure the similarity between two hash codes. The smaller the Hamming distance, the more similar the two hash codes are, that is, the samples corresponding to them may also be more similar in the original data space.

[0159] For candidate samples, they are all data samples in the database. During the retrieval process, the similarity between the sample to be retrieved and the hash code of each candidate sample is calculated, and a candidate sample is determined to be of interest to the user based on the similarity result.

[0160] The target sorting rule refers to the rule used to determine the sorting order of candidate samples during the search process; the target number can refer to the number of target search results that the user hopes to retrieve, or the number of target search results set by the backend personnel to be returned. The target sorting rule and target number can be set according to the actual application situation, and the embodiments of the present application do not limit this.

[0161] In a specific implementation, the retrieval steps may be: first, select the sample to be retrieved, then extract features according to the feature extraction method corresponding to the retrieval sample, obtain the hash code of the retrieval sample through the hash function of the modality corresponding to the retrieval sample, and calculate the Hamming distance between the hash code of the retrieval sample and the hash code of the target retrieval modality in the database. A small Hamming distance represents a high similarity with the retrieval sample. After the Hamming distance is sorted from small to large, the corresponding sample ranked first in the database can be used as the retrieval result.

[0162] Step S107 shown in the embodiment of the present application achieves fast, accurate and real-time cross-modal retrieval by storing the target multimodal hash function and the total target hash code in a database, and dynamically retrieving the samples to be retrieved based on the target multimodal hash function and the total target hash code that are continuously updated in the database.

[0163] Steps S101 to S107 shown in the embodiment of the present application are as follows: obtaining a multimodal dataset; the multimodal dataset includes a paired dataset and an unpaired dataset; extracting features from the multimodal dataset to obtain a multimodal data feature vector; the multimodal data feature vector includes a paired feature vector and an unpaired feature vector; learning the current paired hash code of the paired dataset based on the paired tag information of the paired dataset to obtain a target paired hash code; learning the current unpaired hash code of the unpaired dataset based on the intra-modal similarity matrix between the paired dataset and the unpaired dataset and the inter-modal similarity matrix of the unpaired dataset to obtain a target unpaired hash code; integrating the target paired hash code and the target unpaired hash code based on the total target function to obtain a total target hash code; learning the current multimodal hash function of the multimodal dataset based on the total target hash code and the multimodal data feature vector to obtain a target multimodal hash function, and storing the target multimodal hash function and the total target hash code in a database; retrieving the sample to be retrieved based on the target multimodal hash function and the total target hash code in the database to obtain a target retrieval result. The embodiment of the present application obtains the target paired hash code by learning the current paired hash code using the paired label information of the paired data set, thereby maintaining the similarity and semantic consistency of the paired data set in the hash space, and providing a reliable benchmark for subsequent cross-modal retrieval; the target unpaired hash code is obtained by learning the intra-modal similarity matrix between the paired data set and the unpaired data set and the inter-modal similarity matrix of the unpaired data set, which not only utilizes the intra-modal similarity of the paired data and the unpaired data, but also solves the unpaired data problem by utilizing the semantic information between the paired data and the unpaired data of different modalities, fully utilizing the information in the unpaired data, and compensating for the problem of insufficient paired information by constructing a similarity matrix, thereby improving the generalization ability and robustness of the hash code; based on the overall goal The function integrates the target hash codes of paired data and unpaired data to obtain the total target hash code, which combines the information of paired data and unpaired data so that the final hash code can simultaneously reflect the characteristics and relationships of the two types of data, thereby improving the accuracy and efficiency of cross-modal retrieval; based on the total target hash code and the multimodal data feature vector, the current multimodal hash function of the multimodal dataset is learned to obtain the target multimodal hash function, so that the hash function can be applied to data of multiple modalities and can capture the semantic relationship between different modalities, providing data support for cross-modal retrieval; finally, the samples to be retrieved can be dynamically retrieved based on the continuously updated target multimodal hash function and the total target hash code in the database, thereby achieving fast, accurate and real-time cross-modal retrieval.

[0164] In order to explain the principle of the technical solution of the present invention in detail, the overall process of the present invention is described below in combination with some specific embodiments. It is easy to understand that the following is an explanation of the technical principle of the present invention and cannot be regarded as a limitation of the present invention.

[0165] The embodiment of the present application proposes a semi-paired cross-modal hash retrieval method in a dynamic data environment, that is, a dynamic cross-modal hash retrieval based on semi-paired data, which can perform cross-modal retrieval as close to the real data environment as possible. The SPI H method provided in the embodiment of the present application makes full use of the semantic information of paired data of different modalities to learn the hash code of paired data. At the same time, the SPI H method uses the semantic connection between unpaired data and paired data to learn the hash codes of unpaired data in different modalities respectively. After using the above information to construct the target hash code, after multiple rounds of optimization iterations, the hash code corresponding to each modality at time t can be obtained. That is, when new unpaired data arrives at a multimodal data set, the inter-modal similarity relationship and the intra-modal similarity relationship between paired data and unpaired data are respectively constructed, the semantic labels of the data are fully utilized, and the adaptive margin factor and the orthogonal real-valued matrix are introduced to reduce the quantization loss and speed up the optimization iteration of the algorithm, and the hash codes of each modality of the new data and other related matrices are obtained through optimization iteration, and then the hash code of each modality is calculated for the hash function corresponding to its modality for testing.

[0166] See also Figure 2 , Figure 2 is a flow chart of hash coding learning based on semi-paired data provided in an embodiment of the present application, such as Figure 2 As shown in Fig. 1, the overall process of hash coding learning based on semi-paired data is as follows: before the arrival of new data, the similarity matrices of paired data and unpaired data are constructed according to the paired data label information of the old data (or called old data) and the unpaired data label information of the old data, and an adaptive margin is added to the paired data label information of the old data. Then, the total objective function is constructed, and each objective function in the total objective function (i.e. Figure 2 The semantic embedding step in the above example can obtain the hash codes corresponding to the paired data and unpaired data of the old data. When new data arrives, the similarity matrices of paired data and unpaired data are constructed respectively for the paired data label information of the new data and the unpaired data label information of the new data. Similarly, an adaptive margin is added to the paired data label information of the new data, and then semantic embedding is performed to update the hash code of the data set.

[0167] The SPI H method is a hash method with two-step training. In the specific implementation, the training steps can be: In the first step, the SPI H method makes full use of the semantic information of paired data of different modalities to learn the hash code of paired data. At the same time, the SPI H method uses the semantic connection between unpaired data and paired data to learn the hash code of unpaired data in different modalities respectively. After using the above information to construct the target hash code, after multiple rounds of optimization iterations, the hash code corresponding to each modality at time t can be obtained. In the second step, using the hash code obtained in the first step, the SPI H method uses linear regression to learn the projection (hash function) of each modality.

[0168] Specifically, the retrieval step can be: first, prepare a multimodal data set before retrieval, and extract each modal feature vector of the multimodal data set and the label matrix corresponding to the sample and other information through the corresponding feature extraction method, perform hash coding and hash function learning through each modal feature vector and the label matrix corresponding to the sample and other information to obtain the target hash code and target hash function, and then store the target hash code and target hash function in the database and wait for retrieval. Next, select the retrieval sample, extract features according to the feature extraction method corresponding to the retrieval sample, process the extracted features through the hash function of the modality corresponding to the retrieval sample to obtain the hash code of the retrieval sample, calculate the Hamming distance between the hash code of the retrieval sample and the hash code of the target retrieval modality in the database, the smaller the Hamming distance, the greater the similarity with the sample, and after the Hamming distance is sorted from small to large, the corresponding sample in the front can be used as the retrieval result.

[0169] Specifically, the retrieval step can be divided into the following three steps:

[0170] (1) When new unpaired data arrives at the multimodal dataset, the inter-modal similarity relationship and intra-modal similarity relationship between the paired data and the unpaired data are constructed respectively, making full use of the semantic labels of the data. In addition, an adaptive margin factor and an orthogonal real-valued matrix are introduced to reduce the quantization loss and speed up the optimization iteration of the algorithm.

[0171] (2) The hash codes of each mode of the new data and other related matrices are obtained by optimizing the iterations, and then the hash functions corresponding to the hash codes of each mode are calculated for testing.

[0172] (3) In the testing phase, feature extraction technology is first used to extract the feature vector of the test modality. Then, the hash function is used to calculate the corresponding hash code. The Hamming distance between the hash code of the test data and the hash code of the database is compared. The smaller the distance, the more similar it is to the database sample. The samples are sorted from small to large according to the Hamming distance. The first few samples output are the query results corresponding to the test sample.

[0173] It should be pointed out that this embodiment only briefly illustrates the general process of the dynamic cross-modal hash retrieval method based on semi-paired data. The detailed description of each step can refer to the relevant content in the aforementioned embodiment, which will not be repeated here. It can be understood that the present invention is not limited to this.

[0174] The embodiment of the present application obtains a multimodal dataset; the multimodal dataset includes a paired dataset and an unpaired dataset; feature extraction is performed on the multimodal dataset to obtain a multimodal data feature vector; the multimodal data feature vector includes a paired feature vector and an unpaired feature vector; the current paired hash code of the paired dataset is learned based on the paired label information of the paired dataset to obtain a target paired hash code; the current unpaired hash code of the unpaired dataset is learned based on the intra-modal similarity matrix between the paired dataset and the unpaired dataset and the inter-modal similarity matrix of the unpaired dataset to obtain a target unpaired hash code; the target paired hash code and the target unpaired hash code are integrated based on the total target function to obtain a total target hash code; the current multimodal hash function of the multimodal dataset is learned based on the total target hash code and the multimodal data feature vector to obtain a target multimodal hash function, and the target multimodal hash function and the total target hash code are stored in a database; the sample to be retrieved is retrieved based on the target multimodal hash function and the total target hash code in the database to obtain a target retrieval result. The embodiment of the present application obtains the target paired hash code by learning the current paired hash code using the paired label information of the paired data set, thereby maintaining the similarity and semantic consistency of the paired data set in the hash space, and providing a reliable benchmark for subsequent cross-modal retrieval; the target unpaired hash code is obtained by learning the intra-modal similarity matrix between the paired data set and the unpaired data set and the inter-modal similarity matrix of the unpaired data set, which not only utilizes the intra-modal similarity of the paired data and the unpaired data, but also solves the unpaired data problem by utilizing the semantic information between the paired data and the unpaired data of different modalities, fully utilizing the information in the unpaired data, and compensating for the problem of insufficient paired information by constructing a similarity matrix, thereby improving the generalization ability and robustness of the hash code; based on the overall goal The function integrates the target hash codes of paired data and unpaired data to obtain the total target hash code, which combines the information of paired data and unpaired data so that the final hash code can simultaneously reflect the characteristics and relationships of the two types of data, thereby improving the accuracy and efficiency of cross-modal retrieval; based on the total target hash code and the multimodal data feature vector, the current multimodal hash function of the multimodal dataset is learned to obtain the target multimodal hash function, so that the hash function can be applied to data of multiple modalities and can capture the semantic relationship between different modalities, providing data support for cross-modal retrieval; finally, the samples to be retrieved can be dynamically retrieved based on the continuously updated target multimodal hash function and the total target hash code in the database, thereby achieving fast, accurate and real-time cross-modal retrieval.

[0175] In summary, the key points of the embodiments of the present application are:

[0176] 1. A method for learning the intra-modal similarity relationship and inter-modal similarity relationship of paired data and unpaired data in dynamic and semi-paired data environments respectively.

[0177] 2. An optimized iterative method for solving the final hash function and the final hash code in the embodiment of the present application.

[0178] See also Figure 3 The embodiment of the present application further provides a dynamic cross-modal hash retrieval system 300 based on semi-paired data, which can implement the above-mentioned dynamic cross-modal hash retrieval method based on semi-paired data. The system 300 includes the following modules:

[0179] The data acquisition module 301 is used to acquire a multimodal data set; the multimodal data set includes a paired data set and an unpaired data set;

[0180] A feature extraction module 302 is used to extract features from the multimodal data set to obtain a multimodal data feature vector; the multimodal data feature vector includes a paired feature vector and an unpaired feature vector;

[0181] A pairing hash code learning module 303 is used to learn the current pairing hash code of the pairing data set based on the pairing tag information of the pairing data set to obtain a target pairing hash code;

[0182] An unpaired hash code learning module 304 is used to learn the current unpaired hash code of the unpaired data set based on the intra-modality similarity matrix between the paired data set and the unpaired data set and the inter-modality similarity matrix of the unpaired data set to obtain a target unpaired hash code;

[0183] A total target hash code acquisition module 305 is used to integrate the target paired hash code and the target unpaired hash code based on the total target function to obtain a total target hash code;

[0184] A hash function learning module 306 is used to learn the current multimodal hash function of the multimodal data set based on the total target hash code and the multimodal data feature vector to obtain a target multimodal hash function, and store the target multimodal hash function and the total target hash code in a database;

[0185] The sample retrieval module 307 is used to retrieve the sample to be retrieved based on the target multimodal hash function and the total target hash code in the database to obtain a target retrieval result.

[0186] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0187] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned dynamic cross-modal hash retrieval method based on semi-paired data when executing the computer program. The electronic device can be any smart terminal including a tablet computer, a car computer, etc.

[0188] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0189] See also Figure 4 , Figure 4 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0190] The processor 401 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0191] The memory 402 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 402 can store an operating system and other applications. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 402, and the processor 401 calls and executes the dynamic cross-modal hash retrieval method based on semi-paired data in the embodiment of the present application;

[0192] Input / output interface 403, used to implement information input and output;

[0193] Communication interface 404, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WI FI, Bluetooth, etc.);

[0194] Bus 405 , which transmits information between various components of the device (e.g., processor 401 , memory 402 , input / output interface 403 , and communication interface 404 );

[0195] The processor 401 , the memory 402 , the input / output interface 403 and the communication interface 404 are connected to each other in communication within the device via the bus 405 .

[0196] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned dynamic cross-modal hash retrieval method based on semi-paired data.

[0197] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0198] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0199] The embodiments of the present application provide a dynamic cross-modal hash retrieval method based on semi-paired data and a dynamic cross-modal hash retrieval system based on semi-paired data, which obtain a multimodal data set; the multimodal data set includes a paired data set and an unpaired data set; feature extraction is performed on the multimodal data set to obtain a multimodal data feature vector; the multimodal data feature vector includes a paired feature vector and an unpaired feature vector; based on the paired label information of the paired data set, the current paired hash code of the paired data set is learned to obtain a target paired hash code; based on the intra-modal similarity matrix between the paired data set and the unpaired data set and the unpaired data The inter-modal similarity matrix of the set is used to learn the current unpaired hash code of the unpaired data set to obtain the target unpaired hash code; the target paired hash code and the target unpaired hash code are integrated based on the total target function to obtain the total target hash code; based on the total target hash code and the multimodal data feature vector, the current multimodal hash function of the multimodal data set is learned to obtain the target multimodal hash function, and the target multimodal hash function and the total target hash code are stored in the database; the samples to be retrieved are retrieved based on the target multimodal hash function and the total target hash code in the database to obtain the target retrieval results. The embodiment of the present application obtains the target paired hash code by learning the current paired hash code using the paired label information of the paired data set, thereby maintaining the similarity and semantic consistency of the paired data set in the hash space, and providing a reliable benchmark for subsequent cross-modal retrieval; the target unpaired hash code is obtained by learning the intra-modal similarity matrix between the paired data set and the unpaired data set and the inter-modal similarity matrix of the unpaired data set, which not only utilizes the intra-modal similarity of the paired data and the unpaired data, but also solves the unpaired data problem by utilizing the semantic information between the paired data and the unpaired data of different modalities, fully utilizing the information in the unpaired data, and compensating for the problem of insufficient paired information by constructing a similarity matrix, thereby improving the generalization ability and robustness of the hash code; based on the overall goal The function integrates the target hash codes of paired data and unpaired data to obtain the total target hash code, which combines the information of paired data and unpaired data so that the final hash code can simultaneously reflect the characteristics and relationships of the two types of data, thereby improving the accuracy and efficiency of cross-modal retrieval; based on the total target hash code and the multimodal data feature vector, the current multimodal hash function of the multimodal dataset is learned to obtain the target multimodal hash function, so that the hash function can be applied to data of multiple modalities and can capture the semantic relationship between different modalities, providing data support for cross-modal retrieval; finally, the samples to be retrieved can be dynamically retrieved based on the continuously updated target multimodal hash function and the total target hash code in the database, thereby achieving fast, accurate and real-time cross-modal retrieval.

[0200] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0201] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0202] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0203] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0204] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0205] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0206] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0207] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0208] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0209] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0210] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A dynamic cross-modal hash retrieval method based on semi-paired data, characterized in that: The method comprises the following steps: Acquire a multimodal dataset; the multimodal dataset includes a paired dataset and an unpaired dataset; Performing feature extraction on the multimodal data set to obtain a multimodal data feature vector; the multimodal data feature vector includes a paired feature vector and an unpaired feature vector; Learning the current pairing hash code of the pairing data set based on the pairing label information of the pairing data set to obtain a target pairing hash code; Based on the intra-modality similarity matrix between the paired data set and the unpaired data set and the inter-modality similarity matrix of the unpaired data set, learning the current unpaired hash code of the unpaired data set to obtain a target unpaired hash code; Based on the overall objective function, the target paired hash code and the target unpaired hash code are integrated to obtain an overall target hash code; Learning the current multimodal hash function of the multimodal data set based on the total target hash code and the multimodal data feature vector to obtain a target multimodal hash function, and storing the target multimodal hash function and the total target hash code in a database; The samples to be retrieved are retrieved based on the target multimodal hash function and the total target hash code in the database to obtain a target retrieval result.

2. The method according to claim 1, characterized in that The step of extracting features from the multimodal data set to obtain a multimodal data feature vector includes: Performing a first feature extraction process on the paired data set in the multimodal data set to obtain the paired feature vector; A second feature extraction process is performed on the unpaired data set in the multimodal data set to obtain the unpaired feature vector.

3. The method according to claim 1, characterized in that The learning of the current pairing hash code of the pairing data set based on the pairing tag information of the pairing data set to obtain a target pairing hash code includes: Constructing a pairing label matrix of the pairing data set based on the pairing label information of the pairing data set; Normalizing the paired label matrix to obtain a normalized paired label matrix; Calculating the pairwise semantic similarity between the label vectors in the normalized paired label matrix to obtain the pairwise semantic similarity value; Constructing a pairwise similarity matrix based on the pairwise semantic similarity values; The current pairing hash code of the pairing data set is learned based on the pairing similarity matrix to obtain the target pairing hash code.

4. The method according to claim 1, characterized in that: The integrating the target paired hash code and the target unpaired hash code based on the overall objective function to obtain an overall target hash code includes: Merge the target paired hash code and the target unpaired hash code to obtain a target merged hash code; The target merged hash code is optimized based on the overall objective function to obtain the overall target hash code.

5. The method according to claim 1, characterized in that The learning of the current multimodal hash function of the multimodal data set based on the total target hash code and the multimodal data feature vector to obtain a target multimodal hash function, and storing the target multimodal hash function and the total target hash code in a database includes: Based on the total target hash code and the multimodal data feature vector, the current multimodal hash function of the multimodal data set is learned using a linear regression method to obtain the target multimodal hash function, and the target multimodal hash function and the total target hash code are stored in the database; wherein the target multimodal hash function is used to characterize the target hash function corresponding to various modal data in the multimodal data set.

6. The method according to claim 1, characterized in that The step of retrieving the sample to be retrieved based on the target multimodal hash function and the total target hash code in the database to obtain a target retrieval result includes: Converting the to-be-retrieved feature vector of the to-be-retrieved sample into a to-be-retrieved hash code based on the target multimodal hash function in the database; Calculating the Hamming distance between the to-be-retrieved hash code and the total target hash code in the database to obtain a similarity calculation result; The candidate samples in the database are retrieved according to the similarity calculation result to obtain the target retrieval result.

7. The method according to claim 6, characterized in that The step of retrieving the candidate samples in the database according to the similarity calculation result to obtain the target retrieval result includes: Sorting the candidate samples in the database by similarity according to the similarity calculation result to obtain a similarity ranking result; The candidate samples in the similarity ranking results that meet the target sorting rule and the target quantity are used as the target retrieval results.

8. A dynamic cross-modal hash retrieval system based on semi-paired data, characterized in that: The system includes the following modules: A data acquisition module, used to acquire a multimodal data set; the multimodal data set includes a paired data set and an unpaired data set; A feature extraction module, used to extract features from the multimodal data set to obtain a multimodal data feature vector; the multimodal data feature vector includes a paired feature vector and an unpaired feature vector; A pairing hash code learning module, used to learn the current pairing hash code of the pairing data set based on the pairing label information of the pairing data set to obtain a target pairing hash code; an unpaired hash code learning module, configured to learn a current unpaired hash code of the unpaired data set based on an intra-modality similarity matrix between the paired data set and the unpaired data set and an inter-modality similarity matrix of the unpaired data set to obtain a target unpaired hash code; A total target hash code acquisition module is used to integrate the target paired hash code and the target unpaired hash code based on the total target function to obtain a total target hash code; A hash function learning module, configured to learn a current multimodal hash function of the multimodal data set based on the total target hash code and the multimodal data feature vector to obtain a target multimodal hash function, and store the target multimodal hash function and the total target hash code in a database; The sample retrieval module is used to retrieve the sample to be retrieved based on the target multimodal hash function and the total target hash code in the database to obtain a target retrieval result.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images