Dynamic cross-modal hash retrieval method and system based on concept vector learning

Abstract

The invention discloses a dynamic cross-modal Hash retrieval method and system based on concept vector learning. The method comprises the following steps: constructing a concept vector matrix based on current label information and a Hadamard matrix; concept vectors in the concept vector matrix are used for representing that the samples with the same label have the same and unchanged vectors; constructing a data similarity matrix based on the current label information and the historical label information; obtaining a target hash code based on the concept vector matrix and the data similarity matrix; obtaining a target hash function based on the target hash code; analyzing the to-be-queried data according to the target hash function to obtain a to-be-queried hash code corresponding to the to-be-queried data; and performing similarity calculation on the to-be-queried hash code and a target hash code in a database to obtain a target retrieval result. According to the method, the defect of the concept drift problem in the dynamic data environment can be overcome, the adaptability of the cross-modal Hash model to the concept drift is improved, the dynamic cross-modal Hash retrieval performance is improved, and the method can be widely applied to the technical field of information processing.

Description

Technical Field

[0001] The present application relates to the field of information processing technology, and in particular to a dynamic cross-modal hash retrieval method and system based on concept vector learning. Background Art

[0002] In the field of cross-modal hashing, current cross-modal hashing is difficult to adapt to the concept drift problem in dynamic data environments, and the concept drift problem in dynamic data environments will lead to a decline in the performance of cross-modal hashing.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0004] The embodiments of the present application aim to solve at least one of the technical problems in the related art to a certain extent. To this end, the main purpose of the embodiments of the present application is to propose a dynamic cross-modal hash retrieval method and system based on concept vector learning, which can overcome the defects of the concept drift problem in a dynamic data environment, improve the adaptability of the cross-modal hash model to concept drift, and improve the dynamic cross-modal hash retrieval performance.

[0005] To achieve the above purpose, an embodiment of the present application proposes a dynamic cross-modal hash retrieval method based on concept vector learning, which includes the following steps:

[0006] Obtain current label information of current multimodal data and historical label information of historical multimodal data;

[0007] Based on the current label information and the Hadamard matrix, a concept vector matrix is ​​constructed; wherein the concept vectors in the concept vector matrix are used to characterize that samples with the same label have the same and unchanged vectors;

[0008] Based on the current tag information and the historical tag information, construct a data similarity matrix;

[0009] Performing hash code learning based on the concept vector matrix and the data similarity matrix to obtain a target hash code, and storing the target hash code in a database;

[0010] Performing hash function learning based on the target hash code to obtain a current hash function, and dynamically updating a historical hash function according to the current hash function to obtain a target hash function;

[0011] Analyze the data to be queried according to the target hash function to obtain a hash code to be queried corresponding to the data to be queried;

[0012] The similarity between the hash code to be queried and the target hash code in the database is calculated to obtain a target retrieval result.

[0013] In some embodiments, constructing a concept vector matrix based on the current label information and the Hadamard matrix includes:

[0014] Based on the current label information, assigning the column vector of the Hadamard matrix as a prototype code to a number of current category data in the current multimodal data;

[0015] Based on the current sample label information corresponding to the current sample data in each of the current category data, generating a concept vector corresponding to each of the current sample label information in the concept space;

[0016] The concept vector matrix is ​​constructed based on each of the concept vectors.

[0017] In some embodiments, generating a concept vector corresponding to each current sample label information in a concept space based on the current sample label information corresponding to each current sample data in the current category data includes:

[0018] Based on the current sample label information corresponding to the current sample data in each of the current category data, the concept vector corresponding to each of the current sample label information is generated in the concept space through a concept vector calculation formula; the concept vector calculation formula is:

[0019]

[0020] Among them, F i Represented as a concept vector, c i It is expressed as the number of prototype codes corresponding to the i-th sample data. It represents the jth prototype code of the i-th sample data, i represents the sequence number of the sample data, and j represents the sequence number of the prototype code.

[0021] In some embodiments, the data similarity matrix includes a first pairwise similarity matrix and a second pairwise similarity matrix, and constructing the data similarity matrix based on the current tag information and the historical tag information includes:

[0022] Based on the current label information, calculating first similarity information between current sample data in the current multimodal data, and constructing the first pairwise similarity matrix according to the first similarity information;

[0023] Based on the current label information and the historical label information, second similarity information between the current sample data in the current multimodal data and the historical sample data in the historical multimodal data is calculated, and the second pairwise similarity matrix is ​​constructed according to the second similarity information.

[0024] In some embodiments, the performing hash code learning based on the concept vector matrix and the data similarity matrix to obtain a target hash code, and storing the target hash code in a database includes:

[0025] Constructing a target optimization function based on the concept vector matrix and the data similarity matrix;

[0026] Initialize the current hash code;

[0027] Alternately optimizing the current hash code by using the target optimization function to obtain the target hash code;

[0028] The target hash code is stored in the database.

[0029] In some embodiments, the performing hash function learning based on the target hash code to obtain a current hash function, and dynamically updating a historical hash function according to the current hash function to obtain a target hash function, includes:

[0030] Based on the target hash code, the current multimodal data and the historical multimodal data, a hash function is learned by using a linear regression method to obtain the current hash function;

[0031] The historical hash function is dynamically updated according to the current hash function to obtain the target hash function.

[0032] In some embodiments, the calculating the similarity between the hash code to be queried and the target hash code in the database to obtain the target search result includes:

[0033] Calculating the Hamming distance between the query hash code and the target hash code in the database to obtain a similarity calculation result;

[0034] The target search result is obtained according to the similarity calculation result and the preset search rule.

[0035] To achieve the above purpose, another aspect of the embodiment of the present application proposes a dynamic cross-modal hash retrieval system based on concept vector learning, and the system includes the following modules:

[0036] A data acquisition module is used to acquire current label information of current multimodal data and historical label information of historical multimodal data;

[0037] A concept vector matrix construction module is used to construct a concept vector matrix based on the current label information and the Hadamard matrix; wherein the concept vectors in the concept vector matrix are used to characterize that samples with the same label have the same and unchanged vectors;

[0038] A similarity matrix construction module, used to construct a data similarity matrix based on the current tag information and the historical tag information;

[0039] A hash code learning module, used for performing hash code learning based on the concept vector matrix and the data similarity matrix to obtain a target hash code, and storing the target hash code in a database;

[0040] A hash function learning module, configured to obtain a current hash function based on the target hash code, and dynamically update a historical hash function according to the current hash function to obtain a target hash function;

[0041] A hash code analysis module, used to analyze the data to be queried according to the target hash function to obtain a hash code to be queried corresponding to the data to be queried;

[0042] The hash search module is used to calculate the similarity between the hash code to be queried and the target hash code in the database to obtain a target search result.

[0043] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0044] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0045] The embodiments of the present application include at least the following beneficial effects: the present application provides a dynamic cross-modal hash retrieval method and system based on concept vector learning, the scheme obtains current label information of current multimodal data and historical label information of historical multimodal data; constructs a concept vector matrix based on the current label information and the Hadamard matrix; wherein the concept vectors in the concept vector matrix are used to characterize that samples with the same label have the same and unchanged vectors; constructs a data similarity matrix based on the current label information and the historical label information; performs hash code learning based on the concept vector matrix and the data similarity matrix to obtain a target hash code, and stores the target hash code in a database; performs hash function learning based on the target hash code to obtain a current hash function, and dynamically updates the historical hash function according to the current hash function to obtain a target hash function; analyzes the data to be queried according to the target hash function to obtain a hash code to be queried corresponding to the data to be queried; performs similarity calculation between the hash code to be queried and the target hash code in the database to obtain a target retrieval result. The embodiment of the present application defines the same and unchanged concept vector for samples with the same label in the constructed concept space, and the concept vector will not change with the occurrence of concept drift. Then, the information of the concept vector is learned when learning the hash code, so that the model has the ability to cope with concept drift, overcomes the defect that the related cross-modal hash cannot cope with the concept drift problem in the dynamic data environment, improves the adaptability of the cross-modal hash model to concept drift, and can still efficiently learn hash functions and hash codes in the dynamic data environment where concept drift frequently occurs, thereby improving the dynamic cross-modal hash retrieval performance; at the same time, it also effectively learns the pairwise similarity matrix corresponding to the current multimodal data and the historical multimodal data, and the hash code obtained based on the pairwise similarity matrix and the concept vector matrix can retain the similarity information between a large number of samples, so that the hash code has greater distinguishability, and after obtaining the hash code, the hash function can be effectively and accurately updated using the new and old data information and the hash code, thereby improving the efficiency and accuracy of dynamic cross-modal hash retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a flowchart of the steps of a dynamic cross-modal hash retrieval method based on concept vector learning provided in an embodiment of the present application;

[0047] Figure 2 It is a schematic diagram of the framework of concept vector learning provided in the embodiment of the present application;

[0048] Figure 3 It is a flowchart of a dynamic cross-modal hash retrieval method based on concept vector learning provided in an embodiment of the present application;

[0049] Figure 4 It is a schematic diagram of the framework of online concept vector hashing provided by an embodiment of the present application;

[0050] Figure 5 It is a structural diagram of a dynamic cross-modal hash retrieval system based on concept vector learning provided in an embodiment of the present application;

[0051] Figure 6 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of systems and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0053] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0054] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0056] In the field of cross-modal hashing, current cross-modal hashing is difficult to adapt to the concept drift problem in dynamic data environments, and the concept drift problem in dynamic data environments will lead to a decline in the performance of cross-modal hashing.

[0057] In view of this, a dynamic cross-modal hash retrieval method and system based on concept vector learning are provided in an embodiment of the present application. The scheme obtains current label information of current multimodal data and historical label information of historical multimodal data; constructs a concept vector matrix based on the current label information and the Hadamard matrix; wherein the concept vectors in the concept vector matrix are used to characterize that samples with the same label have the same and unchanged vectors; constructs a data similarity matrix based on the current label information and the historical label information; performs hash code learning based on the concept vector matrix and the data similarity matrix to obtain a target hash code, and stores the target hash code in a database; performs hash function learning based on the target hash code to obtain a current hash function, and dynamically updates the historical hash function according to the current hash function to obtain a target hash function; analyzes the data to be queried according to the target hash function to obtain a hash code to be queried corresponding to the data to be queried; performs similarity calculation between the hash code to be queried and the target hash code in the database to obtain a target retrieval result. The embodiment of the present application defines the same and unchanged concept vector for samples with the same label in the constructed concept space, and the concept vector will not change with the occurrence of concept drift. Then, the information of the concept vector is learned when learning the hash code, so that the model has the ability to cope with concept drift, overcomes the defect that the related cross-modal hash cannot cope with the concept drift problem in the dynamic data environment, improves the adaptability of the cross-modal hash model to concept drift, and can still efficiently learn hash functions and hash codes in the dynamic data environment where concept drift frequently occurs, thereby improving the dynamic cross-modal hash retrieval performance; at the same time, it also effectively learns the pairwise similarity matrix corresponding to the current multimodal data and the historical multimodal data, and the hash code obtained based on the pairwise similarity matrix and the concept vector matrix can retain the similarity information between a large number of samples, so that the hash code has greater distinguishability, and after obtaining the hash code, the hash function can be effectively and accurately updated using the new and old data information and the hash code, thereby improving the efficiency and accuracy of dynamic cross-modal hash retrieval.

[0058] The dynamic cross-modal hash retrieval method based on concept vector learning provided in the embodiment of the present application relates to the field of information processing technology. The dynamic cross-modal hash retrieval method based on concept vector learning provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network, content distribution network) and cloud servers for basic cloud computing services such as big data and artificial intelligence platforms, and the server can also be a node server in a blockchain network; the software can be an application that implements a dynamic cross-modal hash retrieval method based on concept vector learning, etc., but is not limited to the above forms.

[0059] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs (Personal Computers, personal computers), minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0060] See also Figure 1 , Figure 1 is an optional step flow chart of a dynamic cross-modal hash retrieval method based on concept vector learning provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S107.

[0061] Step S101, obtaining current label information of current multimodal data and historical label information of historical multimodal data;

[0062] Among them, the current multimodal data can be called new data or newly arrived data. The current multimodal data refers to a data set containing multiple types of information collected at the current specific time point. For historical multimodal data, it can also be called old data or arrived data. It is a multimodal data set collected at a certain time point or time period in the past. Compared with the current multimodal data, the historical multimodal data reflects the past state or events. Among them, the historical multimodal data is of great significance for the analysis and processing of subsequent data.

[0063] Among them, multimodal data usually includes data in multiple forms such as text, images, audio, and video, which together describe multiple aspects of an entity or event. For example, on social media, a post may contain a text description (words), a picture (image), and possibly a video or audio clip, which together constitute multimodal data.

[0064] For the current label information, it is the label or annotation information associated with the current multimodal data, which is used to describe or classify the content and characteristics of the current multimodal data. For the historical label information, it is the label or annotation information associated with the historical multimodal data. Similarly, the historical label information is used to describe or classify the content and characteristics of the historical multimodal data.

[0065] Step S101 shown in the embodiment of the present application provides data support for subsequent data processing and analysis by acquiring current label information of current multimodal data and historical label information of historical multimodal data.

[0066] Step S102, constructing a concept vector matrix based on the current label information and the Hadamard matrix; wherein the concept vectors in the concept vector matrix are used to represent that samples with the same label have the same and unchanged vectors;

[0067] Among them, the concept vector matrix is ​​composed of the concept vectors corresponding to all samples in the current multimodal data. The concept vector is used to characterize that samples with the same label have the same and unchanged vector. The space where the concept vector is located is called the concept space. It is easy to understand that regardless of whether the concept drift occurs when the multimodal data arrives, the samples with the same label have the same and unchanged real-valued concept vector. Since samples with the same label share the same concept vector in the concept space, and the concept vector will not change with the occurrence of concept drift, learning the information in the concept vector can enable the system to have a certain ability to cope with concept drift, overcome the defect that the related cross-modal hash retrieval cannot cope with the concept drift problem in the dynamic data environment, and improve the adaptability of the cross-modal hash system to concept drift, so that in the dynamic data environment where concept drift frequently occurs, hash functions and hash codes can still be efficiently learned, and excellent retrieval performance can be achieved.

[0068] It is worth mentioning that in order to solve the problem of concept drift in the field of cross-modal hashing, the embodiment of the present application proposes online concept vector hashing (OCVH for short), that is, the dynamic cross-modal hash retrieval method based on concept vector learning provided in the embodiment of the present application. Concept vector learning can better deal with the concept drift problem faced by cross-modal hashing.

[0069] In some embodiments, step S102 may include: based on the current label information, assigning the column vector of the Hadamard matrix as a prototype code to several current category data in the current multimodal data; based on the current sample label information corresponding to the current sample data in each current category data, generating a concept vector corresponding to each current sample label information in the concept space; and constructing a concept vector matrix based on each concept vector.

[0070] Among them, the Hadamard matrix is ​​a special square matrix whose elements are composed of +1 and -1, and the sum of the products of corresponding elements of any two different rows (or columns) is 0. In an embodiment of the present application, a column of the Hadamard matrix is ​​assigned to each category in the current multimodal data as its prototype code. Since the first column of the Hadamard matrix is ​​a full-1 vector, the allocation starts from the second column of the Hadamard matrix. Exemplarily, assuming that there are currently three categories C1, C2, and C3, since the first column of the Hadamard matrix is ​​a full-1 vector, the allocation starts from the second column, then C1 corresponds to the second column, C2 corresponds to the third column, and C3 corresponds to the fourth column.

[0071] For the prototype code, it is a column vector selected from the Hadamard matrix, and the column vector is a reference code used to represent different categories or labels. By assigning a prototype code to each category, samples can be quickly encoded and classified in the hash space.

[0072] Among them, the current category data refers to each category in the current multimodal data. Specifically, the category refers to the different types in the data set. For example, the category can be different object types, such as "cat", "dog", "bird", etc.; in the specific implementation, it is necessary to assign a column of the Hadamard matrix to each category in the current multimodal data as its prototype code. Among them, the current sample data is each sample in the current multimodal data set. The current sample data contains information from different modalities (such as text, image, etc.), and each sample has a corresponding label information, which is called the current sample label information. Specifically, the label is a way to identify the category. The label is usually the name or code of the category to which the instance in the data set belongs. It is easy to understand that in a multi-label or multi-classification problem, a sample can belong to multiple categories, and each category corresponds to a label. For example, a picture containing a cat and a dog may have two labels, such as "cat" and "dog".

[0073] In some embodiments, based on the current sample label information corresponding to the current sample data in each current category data, generating a concept vector corresponding to each current sample label information in the concept space includes: based on the current sample label information corresponding to the current sample data in each current category data, generating a concept vector corresponding to each current sample label information in the concept space through a concept vector calculation formula; the concept vector calculation formula is:

[0074]

[0075] Among them, F i Represented as a concept vector, c i It is expressed as the number of prototype codes corresponding to the i-th sample data. It represents the jth prototype code of the i-th sample data, i represents the sequence number of the sample data, and j represents the sequence number of the prototype code.

[0076] In the specific implementation, the process of concept vector learning is:

[0077] First, the OCVH provided in the embodiment of the present application allocates a column of the Hadamard matrix as its prototype code for each category of the current multimodal data (since the first column of the Hadamard matrix is ​​a full-1 vector, the allocation starts from the second column), and the calculation formula of the prototype code allocated by the Hadamard matrix is ​​as follows:

[0078] G i =H i+1

[0079] Where H∈{-1, 1} r×r represents the Hadamard matrix; i∈{1,2,3......c t}, i represents the serial number of the category, c trepresents the number of categories at time t; G i Represents the prototype code of the i-th category; H i+1 represents the i+1th column of the Hadamard matrix.

[0080] It should be noted that all i and j in the formula in the embodiments of the present application represent the position order in the corresponding context. For example, if i is 1, 2, or 3, then it is the 1st, 2nd, or 3rd.

[0081] Secondly, the current multimodal data X m The concept vector F of the i-th sample in i The concept vector calculation formula is:

[0082]

[0083] Among them, F i Represented as a concept vector; c i It is expressed as the number of prototype codes corresponding to the i-th sample data, that is, the number of labels of the i-th sample; It represents the jth prototype code of the i-th sample data; i represents the sequence number of the sample data, i∈{1,2,3......n}; j represents the sequence number of the prototype code.

[0084] Optionally, the concept vector matrix F r×n is the current multimodal data X m The concept vectors corresponding to all samples in are composed of, where r represents the length of the hash code and n represents the current multimodal data X m According to the above concept vector calculation formula, no matter whether the multimodal data arrives in time or not, samples with the same label have the same and unchanged real-valued concept vector.

[0085] Furthermore, in order to cope with concept drift, when learning hash codes, the hash code of a sample should be close to the concept vectors of similar samples (including itself) and away from the concept vectors of dissimilar samples. The concept vector optimization formula for coping with concept drift to maintain concept vector similarity is as follows:

[0086]

[0087] Where B∈{-1, 1} r×n Represents the current multimodal data (newly arrived data) X m The corresponding hash code, it should be noted that B is composed of all current multimodal data X m The matrix composed of the corresponding hash codes, B is an r×n matrix, r represents the length of the hash code, and n represents the current multimodal data X mThe number of samples; F represents the concept vector matrix; S represents the pairwise similarity matrix between each sample in the current multimodal data, that is, the similarity matrix between new data, hereinafter referred to as the first pairwise similarity matrix; T represents matrix transpose.

[0088] It should be noted that the above concept vector optimization formula means that the information in the concept vectors of all samples in the current multimodal data is stored in the hash code B. The concept vector optimization formula can also be called a concept vector learning item.

[0089] The OCVH method provided in the embodiment of the present application predefines an identical and unchanging vector for samples with the same label. The vector is called a concept vector, and the space where the concept vector is located is called a concept space. In the embodiment of the present application, when learning hash codes, the OCVH method can make the hash code of the sample close to the area where the concept vectors of similar samples are located in the concept space, and away from the area where the concept vectors of dissimilar samples are located. Since samples with the same label share the same concept vector in the concept space, and the vector does not change with the occurrence of concept drift, the OCVH method provided in the embodiment of the present application has a certain ability to cope with concept drift by learning the information in the concept vector.

[0090] Specifically, see Figure 2 , Figure 2 is a schematic diagram of the framework of concept vector learning provided by the embodiment of the present application; Figure 2 As shown in the figure, before the concept drift occurs, it is assumed that there are two classes in the image modality and the text modality, namely C1 and C2; after the concept drift occurs, the data distribution within the C1 category and the C2 category changes, and a new category C3 also appears. After the concept drift, the hash codes corresponding to the samples with only C1 and C2 labels are still projected to their original corresponding areas in the concept space, while the hash codes corresponding to the samples with C3 labels will be projected to their corresponding new areas. The impact of concept drift has been offset to a certain extent.

[0091] Step S102 shown in the embodiment of the present application defines that samples with the same label share a same concept vector in the concept space, and the concept vector does not change with the occurrence of concept drift. By learning the information in the concept vector, the system can have a certain ability to cope with concept drift, thereby overcoming the defect that the related cross-modal hashing cannot cope with the concept drift problem in a dynamic data environment, and improving the adaptability of the cross-modal hashing system to concept drift, so that in a dynamic data environment where concept drift frequently occurs, hash functions and hash codes can still be efficiently learned and excellent retrieval performance can be achieved.

[0092] Step S103, constructing a data similarity matrix based on the current tag information and the historical tag information;

[0093] The data similarity matrix is ​​used to characterize the similarity between samples. The data similarity matrix is ​​calculated based on the label information. The data similarity matrix includes a first pairwise similarity matrix and a second pairwise similarity matrix. The first pairwise similarity matrix can be represented by S. The first pairwise similarity matrix is ​​the current multimodal data X m The pairwise similarity matrix between each sample in is the similarity matrix between new data; the second pairwise similarity matrix can be used Indicates that the second pairwise similarity matrix is ​​the current multimodal data X m and historical multimodal data The pairwise similarity matrix between each sample in , that is, the similarity matrix between new data and old data.

[0094] In some embodiments, step S103 may include: based on the current label information, calculating the first similarity information between the current sample data in the current multimodal data, and constructing a first pairwise similarity matrix based on the first similarity information; based on the current label information and the historical label information, calculating the second similarity information between the current sample data in the current multimodal data and the historical sample data in the historical multimodal data, and constructing a second pairwise similarity matrix based on the second similarity information.

[0095] For the first similarity information, it is the similarity between each sample in the current multimodal data, which is calculated based on the label information corresponding to the sample in the current multimodal data, and is used to construct the first pairwise similarity matrix. For the second similarity information, it is the similarity between the sample in the current multimodal data and the sample in the historical multimodal data, which is calculated based on the label information corresponding to the sample in the current multimodal data and the sample in the historical multimodal data, and is used to construct the second pairwise similarity matrix.

[0096] In the specific implementation, the specific implementation process of similarity learning between new data (similarity learning between current multimodal data) and similarity learning between new and old data (similarity learning between current multimodal data and historical multimodal data) is as follows:

[0097] (1) Similarity learning between new data:

[0098] The goal of hash code learning is to preserve the similarity relationship between samples. Since samples of different modalities share the same label space, labels are used to calculate the similarity between samples. The initial calculation formula for the pairwise similarity matrix of new data samples is as follows:

[0099]

[0100] Among them, S ijRepresents the pairwise similarity matrix of new data samples (not fine-grained optimization); L i represents the i-th column of the label matrix; L j The jth column of the label matrix; Represents the inner product operator.

[0101] However, since the initial calculation formula of the pairwise similarity matrix of the new data sample cannot meet the fine-grained requirements, a fine-grained pairwise similarity matrix calculation formula is used in the embodiment of the present application to calculate the similarity matrix. The fine-grained calculation formula of the pairwise similarity matrix of the new data sample is as follows:

[0102] S=2V T V-11 T

[0103] Where S represents the pairwise similarity matrix of the new data sample (fine-grained optimization), that is, the first pairwise similarity matrix, V represents the 2-norm normalized label matrix corresponding to the new data, and the i-th column of V is defined as V i =L i / ||L i ||, Represents the current multimodal data (newly arrived data) X m The corresponding label matrix, c t represents the number of categories at time t, and n represents the current multimodal data X m The number of samples; V T V calculates the cosine similarity between sample labels and then maps it to [-1,1] to obtain the pairwise similarity matrix S; T represents the matrix transpose, and 1 represents the all-1 vector.

[0104] Since the Hamming distance between hash codes of similar samples should be as small as possible, and the Hamming distance between hash codes of dissimilar samples should be as large as possible, in order to retain the similarity information in the pairwise similarity matrix B of the new data sample, the following initial new sample similarity retention formula can be used:

[0105]

[0106] Where B∈{-1, 1} r×n Represents the current multimodal data (newly arrived data) X m The corresponding hash code, it should be noted that B is composed of all current multimodal data X m The matrix composed of the corresponding hash codes, B is an r×n matrix, r represents the length of the hash code, and n represents the current multimodal data X m The number of samples; B i represents the i-th column of B; B j represents the jth column of B; at time t, the newly arrived data is m represents different modes and m∈{1,2},d m represents the dimension of m-modal features, and n represents the number of samples of newly arrived data; represents the real-valued space; S ij represents the pairwise similarity matrix of the new data sample (not fine-grained optimization); ||·|| represents the 2-norm of the vector or the Frobenius norm of the matrix.

[0107] Since the above-mentioned initial new sample similarity retention formula has a minimization problem, the embodiment of the present application optimizes the minimization problem to make the similarity information more easily retained in the pairwise similarity matrix B of the new data sample. For the convenience of optimization, the initial new sample similarity retention formula can be simplified to the following target new sample similarity retention formula, which can be called the similarity learning item between new data. The target new sample similarity retention formula is as follows:

[0108]

[0109] Where tr{·} represents the trace function.

[0110] It is easy to understand that the current multimodal data (newly arrived data) X m The similarity information of the target new sample can be retained in the current multimodal data (newly arrived data) X by the above target new sample similarity retention formula m The corresponding hash code.

[0111] (2) Similarity learning between new and old data:

[0112] Current multimodal data (newly arrived data) X m The hash code of the sample in should be consistent with the historical multimodal data (previously reached data) The hash codes of similar samples in the current multimodal data are close, whereas the hash codes of samples in the current multimodal data are far from the hash codes of dissimilar samples in the historical multimodal data. The calculation formula of the pairwise similarity matrix between the new and old data is as follows:

[0113]

[0114] in, represents the pairwise similarity matrix between the new and old data, i.e., the second pairwise similarity matrix; Represents the 2-norm normalized label matrix corresponding to the old data, whose i-th column is Representation and historical multimodal data (previously achieved data) The corresponding label matrix, N represents the number of samples that have arrived before, c trepresents the number of categories at time t, in particular, The number of categories in is the same as the number of categories at time t. If a new category appears at time t, a full 0 matrix is ​​used to replace The dimension is extended to c t ×N; T represents matrix transpose; where, m represents different modes and m∈{1,2},d m represents the dimension of m-modal features, represents a real-valued space.

[0115] Furthermore, in order to preserve the similarity information between the new and old data in the pairwise similarity matrix B of the new data sample, it can be achieved by the following target new and old sample similarity preservation formula, specifically:

[0116]

[0117] in, Representation and historical multimodal data (previously achieved data) The corresponding hash code, N represents the number of samples of data that have arrived before, and r represents the length of the hash code.

[0118] It is worth mentioning that in the subsequent hash code learning, three types of information need to be obtained in advance, namely the similarity information between new data, the information in the concept vector, and the similarity information between new and old data. New data corresponds to the hash code of new data, and old data corresponds to the hash code of old data. At time t, only the hash code B of the new data arriving at time t needs to be learned, and the hash code of the old data has been learned at the previous time.

[0119] Step S103 shown in the embodiment of the present application comprehensively characterizes the similarity between samples in the current multimodal data and the similarity between samples in the current multimodal data and the historical multimodal data by constructing a first pairwise similarity matrix and a second pairwise similarity matrix. The construction of the similarity matrix provides an important data basis for subsequent hash code learning and hash function learning; and, through the similarity retention formula, the similarity information between new data and the similarity information between new and old data are retained in the hash code corresponding to the current multimodal data, providing a data basis for subsequent hash code learning.

[0120] Step S104, performing hash code learning based on the concept vector matrix and the data similarity matrix to obtain a target hash code, and storing the target hash code in a database;

[0121] Among them, the concept vector matrix contains the information in the concept vector, and the data similarity matrix contains the similarity information between new data and the similarity information between new and old data.

[0122] In some embodiments, step S104 may include: constructing a target optimization function based on the concept vector matrix and the data similarity matrix; initializing the current hash code; alternately optimizing the current hash code through the target optimization function to obtain a target hash code; and storing the target hash code in a database.

[0123] The acquisition of the target optimization function is specifically the content of the hash code constraint. The acquisition process of the target optimization function is as follows:

[0124] In the embodiment of the present application, in order to make the hash code contain as much information as possible, it is necessary to maximize the variance of the hash code to obtain higher entropy, which is specifically achieved by the following hash code variance maximization formula:

[0125]

[0126] In addition, the hash code B of the new data needs to be subject to bit balance constraints and bit irrelevant constraints to obtain a higher quality hash code. The overall objective function expression of hash code learning is as follows:

[0127]

[0128] stB1=0,BB T =nE

[0129] Among them, α, β, and γ are all trade-off parameters, B1=0 is used to ensure that the hash code B of the new data is balanced, and BB T =nE is used to ensure that the hash code B of the new data is irrelevant, and n represents the number of samples corresponding to the newly arrived data. It should be noted that n in the embodiments of the present application represents the number of samples corresponding to the newly arrived data, r represents the hash code length, E represents the unit matrix, and 0 represents the all-0 matrix.

[0130] However, since the hash code B of the new data is a discrete value matrix, when these two constraints are imposed on the hash code B of the new data, the overall objective function expression becomes an NP-Hard (Non-deterministic Polynomial-time Hard) problem. Therefore, in the embodiment of the present application, the B on the right side of each term in the overall objective function expression is replaced by a real value matrix To deal with this situation, the constraint on B is relaxed to Y. The final objective optimization function expression of hash code learning is as follows:

[0131]

[0132] st Y1=0,YY T =nE

[0133] It should be noted that in the process of hash code learning, at time t, only the hash code B of the new data arriving at time t needs to be learned; the hash code of the old data The learning has been done at the previous moment, and there is no need to learn the hash code again.

[0134] In the embodiment of the present application, the hash code B of the current new data (ie, the current hash code) is first initialized, and then the final hash code B of the new data (ie, the target hash code) and the real-valued matrix Y are solved using an alternating optimization process.

[0135] The current hash code is initialized by the hash code initialization formula. Since the current hash code does not meet the quality requirements of the target optimization function, it is necessary to optimize the current hash code alternately to obtain the final target hash code B. The process of initializing the current hash code is as follows:

[0136] First, the pairwise similarity matrix S of the new data sample is eigen-decomposed. The eigen-decomposition formula is as follows:

[0137]

[0138] Among them, λ i represents the i-th largest eigenvalue; J i Represents the normalized eigenvector corresponding to the i-th largest eigenvalue; n represents the number of samples of newly arrived data; i represents the bit order. For example, assuming that there are three eigenvalues, namely 1, 2, and 3, the first largest eigenvalue is 3, the second largest eigenvalue is 2, and the third largest eigenvalue is 1.

[0139] Next, initialize the current hash code using the following hash code initialization formula, specifically:

[0140] B=sgn([J2,J3,......J r+1 ] T )

[0141] Among them, sgn represents an element-by-element operator, which is 1 when the element is greater than 0, otherwise it is -1.

[0142] Optionally, the alternating optimization is to optimize the hash code by repeated iterations until a satisfactory solution of the target optimization function is reached to obtain the target hash code. In a specific implementation, the process of performing alternating optimization based on the target optimization function to solve the hash code B of the final new data is to alternately optimize the hash code B of the current new data and the real value matrix Y to solve the final hash code B (i.e., the target hash code) and the real value matrix Y of the new data:

[0143] 1) Update the real-valued matrix Y

[0144] First, when updating the real-valued matrix Y, the hash code B of the new data is fixed. According to the expression of the above objective optimization function, the following real-valued matrix update expression can be obtained, specifically:

[0145]

[0146] stY1=0,YY T =nE

[0147] Next, the calculation formula of the pairwise similarity matrix between new and old data is Substituting into the above real-valued matrix update expression, the real-valued matrix update expression is rewritten as follows. The rewritten formula can be called the target real-valued matrix update expression:

[0148]

[0149] st Y1=0,YY T =nE

[0150] Among them, an auxiliary variable M is defined. Specifically, the expression of the auxiliary variable M is as follows:

[0151]

[0152] Optionally, historical multimodal data (data that has been reached previously) The corresponding hash code and its corresponding 2-norm normalized label matrix are rewritten as and and They represent the hash codes of all data arriving before time t-1 and the corresponding 2-norm normalized label matrix; B t-1 and V t-1 They represent the hash code of the data arriving at time t-1 and the corresponding 2-norm normalized label matrix. Then we can get the following first intermediate matrix recursive formula:

[0153]

[0154] Among them, the intermediate matrix K1 is defined as The intermediate matrix K2 is The above first intermediate matrix recursion formula can be accessed by Intermediate Matrix The auxiliary variable M is calculated with the new data without directly accessing the old data of size N.

[0155] Specifically, define the centralization matrix The intermediate value matrix MDM generated during the solution processT Perform SVD decomposition on it, and the SVD decomposition formula is as follows:

[0156]

[0157] in, represents a positive diagonal matrix, Represents the intermediate value matrix MDM T rank, represents the corresponding eigenvector of a positive eigenvalue, represents the corresponding eigenvector of the remaining eigenvalues. Therefore, the closed-form solution of the above target real-valued matrix update expression can be calculated by the following target real-valued matrix closed-form solution formula:

[0158]

[0159] in, Indicates that through The orthogonal matrix obtained by executing the Gram-Schmidt process, Ψ (no special meaning, an intermediate value generated during the solution process) is calculated as Ψ = DM T IΛ -1 / 2 , represents a random orthogonal matrix. hour, and Is empty.

[0160] 2) Update the hash code B of the new data:

[0161] When the hash code B of the new data is updated, the real-valued matrix Y is fixed, and the real-valued matrix Y at this time is the updated matrix. Therefore, the expression of the above target optimization function can be simplified to the following formula, and the simplified expression of the target optimization function can be called the target solution formula of the hash code of the new data, specifically:

[0162]

[0163] Then the target solution formula based on the hash code of the new data can directly solve the hash code B of the new data. The expression of the hash code B of the solved new data is as follows:

[0164] B=sgn(SY T +γY T )

[0165] At this time, the hash code B of the new data is the updated hash code.

[0166] Step S104 shown in the embodiment of the present application generates a set of optimized hash codes (i.e., target hash codes) by using the concept vector matrix and the data similarity matrix to perform hash code learning. This process combines the hash code constraints and alternating optimization strategies of the target optimization function to ensure that the generated hash codes can accurately reflect the similarity or semantic relationship between the data.

[0167] Step S105, performing hash function learning based on the target hash code to obtain a current hash function, and dynamically updating a historical hash function according to the current hash function to obtain a target hash function;

[0168] In some embodiments, step S105 may include: based on the target hash code, current multimodal data and historical multimodal data, using a linear regression method to learn a hash function to obtain a current hash function; and dynamically updating the historical hash function according to the current hash function to obtain a target hash function.

[0169] For hash function learning, it is a mapping function obtained through training data (including target hash code, current multimodal data and historical multimodal data), which can map the input data sample to a corresponding binary hash code representation. Among them, the mapping process should try to retain the similarity relationship between data samples, that is, the hash codes of similar samples are close in Hamming distance, while the hash codes of dissimilar samples are far apart. In the embodiment of the present application, the linear regression method is used for hash function learning. For the specific implementation principle of the linear regression method, those skilled in the art can refer to the implementation scheme of the relevant technology, and the embodiment of the present application will not be described in detail here.

[0170] Among them, the current hash function refers to the hash function learned based on the target hash code, current multimodal data and historical multimodal data. The current hash function can reflect the distribution and characteristics of the current data and can map new data samples to the corresponding hash code. However, as new data is continuously added, the distribution of data may change, so the current hash function needs to update the historical hash function to adapt to this change. Among them, the historical hash function refers to the previously learned hash function, which is learned based on the data set and hash code that arrived at the previous time.

[0171] The target hash function is obtained by dynamically updating the historical hash function with the current hash function, and is the latest target hash function.

[0172] In the specific implementation, the specific implementation process of hash function learning is as follows:

[0173] After learning the hash code, the OCVH method provided in the embodiment of the present application needs to learn the hash function through the hash code. Specifically, the newly arrived data X m and the newly arrived data Xm The corresponding hash code B is learned through the linear regression method. P m Representing the m-mode hash function, the initial expression of the linear regression learning hash function is as follows:

[0174]

[0175] Wherein, μ represents a trade-off parameter. However, the initial expression of the linear regression learning hash function only considers new data, but not old data. As new data is continuously added, the retrieval performance of the model for old data will inevitably decline. In order to cope with this situation, the embodiment of the present application also needs to consider old data when learning the hash function. The target expression of the optimized linear regression learning hash function is as follows:

[0176]

[0177] Among them, let the above formula (the target expression of linear regression learning hash function) be the hash function P m The partial derivative of is zero, then the hash function P m It can be solved by the following hash function calculation formula, specifically:

[0178]

[0179] Among them, historical multimodal data (data that has arrived before) is rewritten as represents the data that arrived before time t-1, X m t-1 represents the data arriving at time t-1. Then the following recursive formula for the second intermediate matrix can be obtained:

[0180]

[0181] Among them, the intermediate matrix K3 is defined as The intermediate matrix K4 is According to the above second intermediate matrix recursion formula, the hash function P m No need to access old data of size N, only the intermediate matrix Intermediate Matrix and new data.

[0182] In step S105 shown in the embodiment of the present application, by combining the target hash code, current multimodal data and historical multimodal data, a linear regression method is used to learn the hash function to obtain the current hash function, and then the historical hash function is dynamically updated according to the current hash function to obtain a target hash function that is more adaptable to data changes, so that the target hash function can accurately reflect the similarities and differences of the data.

[0183] Step S106, analyzing the data to be queried according to the target hash function to obtain a hash code to be queried corresponding to the data to be queried;

[0184] The data to be queried refers to the original data on which the user or system wishes to perform some operation (such as search, comparison, verification, etc.), and the data to be queried can be any form of information such as text, image, audio, video, etc.

[0185] The hash code to be queried is the output obtained after the data to be queried is processed by the target hash function, and is used for similarity matching with the target hash code stored in the database.

[0186] Step S106 shown in the embodiment of the present application analyzes the data to be queried according to the target hash function to obtain the hash code to be queried corresponding to the data to be queried, so as to quickly locate the data items that may match in the database, and then perform further accurate comparison.

[0187] Step S107: Calculate the similarity between the hash code to be queried and the target hash code in the database to obtain a target search result.

[0188] In some embodiments, step S107 may include: calculating the Hamming distance between the hash code to be queried and the target hash code in the database to obtain a similarity calculation result; and obtaining a target search result according to the similarity calculation result and a preset search rule.

[0189] Among them, similarity calculation refers to the calculation of the Hamming distance between the query hash code and the target hash code in the database. The Hamming distance between similar sample hash codes is small, and the Hamming distance between dissimilar sample hash codes is large.

[0190] By retrieving samples of other modalities that are close to the sample to be queried in the Hamming space, cross-modal retrieval can be efficiently implemented. For example, taking image query text cross-modal retrieval as an example, cross-modal hashing first uses existing samples in the current database to train the model to obtain hash functions and hash codes. The hash codes of image samples and text samples are in the same Hamming space. When executing a query, the hash function projects the image sample to be queried into the Hamming space and obtains its hash code. Then, the Hamming distance between the hash code of the text sample in the database and the hash code of the image sample to be queried is calculated, and the text sample with a smaller Hamming distance is returned as the retrieval result.

[0191] Among them, the preset search rule can be the number of search results outputted, or can be the search results outputted within a certain Hamming distance threshold. The preset search rule can be set according to actual conditions, and the embodiment of the present application does not impose any restrictions on this.

[0192] In the specific retrieval process, assuming that the query data given by modality m is in, Indicates the number of samples to be queried. The corresponding hash code B q The above hash code learning process can be calculated as Finally with B q Samples with small inter-Hamming distances are returned as query results.

[0193] Step S17 shown in the embodiment of the present application can efficiently implement dynamic cross-modal retrieval by searching samples of other modalities that are close to the query sample in the Hamming space.

[0194] Steps S101 to S107 shown in the embodiment of the present application are performed by obtaining current label information of current multimodal data and historical label information of historical multimodal data; constructing a concept vector matrix based on the current label information and the Hadamard matrix; wherein the concept vectors in the concept vector matrix are used to characterize that samples with the same label have the same and unchanged vectors; constructing a data similarity matrix based on the current label information and the historical label information; performing hash code learning based on the concept vector matrix and the data similarity matrix to obtain a target hash code, and storing the target hash code in a database; performing hash function learning based on the target hash code to obtain a current hash function, and dynamically updating the historical hash function according to the current hash function to obtain a target hash function; analyzing the data to be queried according to the target hash function to obtain a hash code to be queried corresponding to the data to be queried; performing similarity calculation between the hash code to be queried and the target hash code in the database to obtain a target retrieval result. The embodiment of the present application defines the same and unchanged concept vector for samples with the same label in the constructed concept space, and the concept vector will not change with the occurrence of concept drift. Then, the information of the concept vector is learned when learning the hash code, so that the model has the ability to cope with concept drift, overcomes the defect that the related cross-modal hash cannot cope with the concept drift problem in the dynamic data environment, improves the adaptability of the cross-modal hash model to concept drift, and can still efficiently learn hash functions and hash codes in the dynamic data environment where concept drift frequently occurs, thereby improving the dynamic cross-modal hash retrieval performance; at the same time, it also effectively learns the pairwise similarity matrix corresponding to the current multimodal data and the historical multimodal data, and the hash code obtained based on the pairwise similarity matrix and the concept vector matrix can retain the similarity information between a large number of samples, so that the hash code has greater distinguishability, and after obtaining the hash code, the hash function can be effectively and accurately updated using the new and old data information and the hash code, thereby improving the efficiency and accuracy of dynamic cross-modal hash retrieval.

[0195] In summary, the OCVH method provided in the embodiment of the present application defines the same and unchanged real-valued concept vector for samples with the same label in the constructed concept space, and learns the information of the concept vector when learning the hash code, so that the dynamic cross-modal hashing has the ability to cope with concept drift; at the same time, the OCVH method effectively learns the pairwise similarity matrix between new data and new data, as well as the pairwise similarity matrix between old data and new data. The hash code obtained based on the pairwise similarity matrix and the concept vector matrix retains the similarity information between samples more completely, so the hash code has good distinguishability; after obtaining the hash code, the OCVH method effectively updates the hash function using the new and old data information and the hash code through linear regression, thereby improving the dynamic cross-modal hash retrieval performance.

[0196] In order to explain the principle of the technical solution of the present invention in detail, the overall process of the present invention is described below in combination with some specific embodiments. It is easy to understand that the following is an explanation of the technical principle of the present invention and cannot be regarded as a limitation of the present invention.

[0197] The algorithm code corresponding to the learning process of the OCVH method provided in the embodiment of the present application at time t can be expressed by the following content, wherein the hyperparameter settings are r=64, α=0.5, β=0.1, γ=0.1, μ=1 and the maximum number of iterations=10, specifically:

[0198] "Input: Newly arrived data X m and the newly arrived data X m The corresponding label matrix L, hash code length r, parameters α, β, γ, μ, auxiliary variables (intermediate matrices) K1, K2, K3, K4 at time t-1;

[0199] 1. Initialize the Hadamard matrix H;

[0200] 2. Expand the dimension of K1 to r×c using a matrix of all zeros t ;

[0201] 3. Initialize the hash code B of the new data according to the hash code initialization formula;

[0202] 4. Enter the loop;

[0203] 5. Update the real-valued matrix Y according to the closed-form solution formula of the target real-valued matrix;

[0204] 6. Update hash code B according to the target solution formula of the hash code of the new data;

[0205] 7. Until convergence or reaching the maximum number of iterations;

[0206] 8. Update the hash function P according to the hash function calculation formula m ;

[0207] 9. Update auxiliary variables K1, K2, K3, K4 according to the first intermediate matrix recursion formula and the second intermediate matrix recursion formula;

[0208] Output: hash code B, hash function P m , updated auxiliary variables K1, K2, K3, K4".

[0209] See also Figure 3 , Figure 3 is a flow chart of a dynamic cross-modal hash retrieval method based on concept vector learning provided in an embodiment of the present application; Figure 3 As shown, Figure 3This is a specific implementation flow chart of the OCVH method proposed in the embodiment of the present application to realize cross-modal retrieval. The overall process of dynamic cross-modal hash retrieval based on concept vector learning is as follows:

[0210] like Figure 3 As shown in the figure, suppose that the top 100 samples of other modalities that are most relevant to the query sample of a certain modality are required to be retrieved from the database. On the one hand, at time t, the image data and text data are first input into the OCVH method. The OCVH method obtains the hash code and updates the hash function of each modality according to the algorithm code corresponding to the learning process at time t. Then, the image data, text data and their corresponding hash codes are stored in the database. For details, please refer to Figure 4 , Figure 4 is a schematic diagram of the framework of online concept vector hashing provided by the embodiment of the present application; Figure 4As shown, the learning process of the OCVH method proposed in the embodiment of the present application is mainly divided into hash code learning and hash function learning. Among them, the hash code learning process is divided into two parts: one is concept vector learning, and the other is similarity learning. Among them, concept vector learning is the focus of the embodiment of the present application. Specifically, the model first assigns a column of the Hadamard matrix as its prototype code for each category of the data at time t according to the label of the data at time t (image data at time t and text data at time t), and then generates its corresponding concept vector in the concept space for each sample of the data at time t according to its label. The concept vectors corresponding to the samples of all the data at time t constitute a concept vector matrix. The OCVH method uses the concept vector matrix to perform concept vector learning to obtain the ability to cope with concept drift. Among them, in the similarity learning process, the OCVH method first calculates the similarity matrix between new data through the labels of the data at time t (image data at time t and text data at time t), and calculates the similarity matrix between new and old data through the labels of the data before time t (image data before time t and text data before time t), and then uses the similarity matrix between new data and the similarity matrix between new and old data to perform similarity learning to retain the similarity information between samples in the hash code. After completing the hash code learning, the OCVH method updates the hash function of the text modality and the hash function of the image modality according to the data at all times (time t and the time before t) and their corresponding hash codes. On the other hand, the retrieval process is specifically as follows: for the queried image, the updated image hash function is used to obtain the queried image hash code corresponding to the queried image, and then the Hamming distance between the queried image hash code and the hash codes of all texts in the database is calculated, and the 100 texts with the smallest Hamming distance to the queried image hash code are returned as retrieval results; similarly, for the queried text, the updated text hash function is used to obtain the queried text hash code corresponding to the queried text, and then the Hamming distance between the queried text hash code and the hash codes of all images in the database is calculated, and the 100 images with the smallest Hamming distance to the queried text hash code are returned as retrieval results.

[0211] It should be pointed out that this embodiment only briefly illustrates the general process of the dynamic cross-modal hash retrieval method based on concept vector learning. The detailed description of each step can refer to the relevant content in the aforementioned embodiment and will not be repeated here. It can be understood that the present invention is not limited to this.

[0212] The embodiment of the present application obtains current label information of current multimodal data and historical label information of historical multimodal data; constructs a concept vector matrix based on the current label information and the Hadamard matrix; wherein the concept vectors in the concept vector matrix are used to characterize that samples with the same label have the same and unchanged vectors; constructs a data similarity matrix based on the current label information and the historical label information; performs hash code learning based on the concept vector matrix and the data similarity matrix to obtain a target hash code, and stores the target hash code in a database; performs hash function learning based on the target hash code to obtain a current hash function, and dynamically updates the historical hash function according to the current hash function to obtain the target hash function; analyzes the data to be queried according to the target hash function to obtain the hash code to be queried corresponding to the data to be queried; performs similarity calculation between the hash code to be queried and the target hash code in the database to obtain a target retrieval result. The embodiment of the present application defines the same and unchanged concept vector for samples with the same label in the constructed concept space, and the concept vector will not change with the occurrence of concept drift. Then, the information of the concept vector is learned when learning the hash code, so that the model has the ability to cope with concept drift, overcomes the defect that the related cross-modal hash cannot cope with the concept drift problem in the dynamic data environment, improves the adaptability of the cross-modal hash model to concept drift, and can still efficiently learn hash functions and hash codes in the dynamic data environment where concept drift frequently occurs, thereby improving the dynamic cross-modal hash retrieval performance; at the same time, it also effectively learns the pairwise similarity matrix corresponding to the current multimodal data and the historical multimodal data, and the hash code obtained based on the pairwise similarity matrix and the concept vector matrix can retain the similarity information between a large number of samples, so that the hash code has greater distinguishability, and after obtaining the hash code, the hash function can be effectively and accurately updated using the new and old data information and the hash code, thereby improving the efficiency and accuracy of dynamic cross-modal hash retrieval.

[0213] See also Figure 5 The embodiment of the present application also provides a dynamic cross-modal hash retrieval system 500 based on concept vector learning, which can implement the above-mentioned dynamic cross-modal hash retrieval method based on concept vector learning. The system 500 includes the following modules:

[0214] The data acquisition module 501 is used to acquire current label information of current multimodal data and historical label information of historical multimodal data;

[0215] A concept vector matrix construction module 502 is used to construct a concept vector matrix based on the current label information and the Hadamard matrix; wherein the concept vectors in the concept vector matrix are used to represent that samples with the same label have the same and unchanged vectors;

[0216] A similarity matrix construction module 503 is used to construct a data similarity matrix based on the current tag information and the historical tag information;

[0217] A hash code learning module 504 is used to perform hash code learning based on the concept vector matrix and the data similarity matrix to obtain a target hash code, and store the target hash code in a database;

[0218] A hash function learning module 505 is used to obtain a current hash function based on the target hash code, and dynamically update a historical hash function according to the current hash function to obtain a target hash function;

[0219] A hash code analysis module 506 is used to analyze the data to be queried according to the target hash function to obtain a hash code to be queried corresponding to the data to be queried;

[0220] The hash search module 507 is used to calculate the similarity between the hash code to be queried and the target hash code in the database to obtain a target search result.

[0221] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0222] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned dynamic cross-modal hash retrieval method based on concept vector learning when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0223] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0224] See also Figure 6 , Figure 6 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0225] The processor 601 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC for short), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0226] The memory 602 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 602 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 602, and the processor 601 calls and executes the dynamic cross-modal hash retrieval method based on concept vector learning in the embodiment of the present application;

[0227] Input / output interface 603, used to implement information input and output;

[0228] Communication interface 604, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WI FI, Bluetooth, etc.);

[0229] Bus 605 , which transmits information between various components of the device (e.g., processor 601 , memory 602 , input / output interface 603 , and communication interface 604 );

[0230] The processor 601 , the memory 602 , the input / output interface 603 and the communication interface 604 are connected to each other in communication within the device via a bus 605 .

[0231] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned dynamic cross-modal hash retrieval method based on concept vector learning.

[0232] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0233] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0234] The embodiments of the present application provide a dynamic cross-modal hash retrieval method based on concept vector learning and a dynamic cross-modal hash retrieval system based on concept vector learning, which obtain the current label information of the current multimodal data and the historical label information of the historical multimodal data; construct a concept vector matrix based on the current label information and the Hadamard matrix; wherein the concept vectors in the concept vector matrix are used to characterize that samples with the same label have the same and unchanged vectors; construct a data similarity matrix based on the current label information and the historical label information; perform hash code learning based on the concept vector matrix and the data similarity matrix to obtain a target hash code, and store the target hash code in a database; perform hash function learning based on the target hash code to obtain a current hash function, and dynamically update the historical hash function according to the current hash function to obtain the target hash function; analyze the data to be queried according to the target hash function to obtain the hash code to be queried corresponding to the data to be queried; perform similarity calculation between the hash code to be queried and the target hash code in the database to obtain a target retrieval result. The embodiment of the present application defines the same and unchanged concept vector for samples with the same label in the constructed concept space, and the concept vector will not change with the occurrence of concept drift. Then, the information of the concept vector is learned when learning the hash code, so that the model has the ability to cope with concept drift, overcomes the defect that the related cross-modal hash cannot cope with the concept drift problem in the dynamic data environment, improves the adaptability of the cross-modal hash model to concept drift, and can still efficiently learn hash functions and hash codes in the dynamic data environment where concept drift frequently occurs, thereby improving the dynamic cross-modal hash retrieval performance; at the same time, it also effectively learns the pairwise similarity matrix corresponding to the current multimodal data and the historical multimodal data, and the hash code obtained based on the pairwise similarity matrix and the concept vector matrix can retain the similarity information between a large number of samples, so that the hash code has greater distinguishability, and after obtaining the hash code, the hash function can be effectively and accurately updated using the new and old data information and the hash code, thereby improving the efficiency and accuracy of dynamic cross-modal hash retrieval.

[0235] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0236] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0237] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0238] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0239] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0240] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0241] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0242] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0243] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0244] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0245] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A dynamic cross-modal hash retrieval method based on concept vector learning, characterized in that: The method comprises the following steps: Obtain current label information of current multimodal data and historical label information of historical multimodal data; Based on the current label information and the Hadamard matrix, a concept vector matrix is ​​constructed; wherein the concept vectors in the concept vector matrix are used to characterize that samples with the same label have the same and unchanged vectors; Based on the current tag information and the historical tag information, construct a data similarity matrix; Performing hash code learning based on the concept vector matrix and the data similarity matrix to obtain a target hash code, and storing the target hash code in a database; Performing hash function learning based on the target hash code to obtain a current hash function, and dynamically updating a historical hash function according to the current hash function to obtain a target hash function; Analyze the data to be queried according to the target hash function to obtain a hash code to be queried corresponding to the data to be queried; The similarity between the hash code to be queried and the target hash code in the database is calculated to obtain a target retrieval result.

2. The method according to claim 1, characterized in that The constructing of a concept vector matrix based on the current label information and the Hadamard matrix includes: Based on the current label information, assigning the column vector of the Hadamard matrix as a prototype code to a number of current category data in the current multimodal data; Based on the current sample label information corresponding to the current sample data in each of the current category data, generating a concept vector corresponding to each of the current sample label information in the concept space; The concept vector matrix is ​​constructed based on each of the concept vectors.

3. The method according to claim 2, characterized in that The generating, based on the current sample label information corresponding to the current sample data in each of the current category data, a concept vector corresponding to each of the current sample label information in the concept space includes: Based on the current sample label information corresponding to the current sample data in each of the current category data, the concept vector corresponding to each of the current sample label information is generated in the concept space through a concept vector calculation formula; the concept vector calculation formula is: Among them, F i Represented as a concept vector, c i It is expressed as the number of prototype codes corresponding to the i-th sample data. It represents the jth prototype code of the i-th sample data, i represents the sequence number of the sample data, and j represents the sequence number of the prototype code.

4. The method according to claim 1, characterized in that: The data similarity matrix includes a first pairwise similarity matrix and a second pairwise similarity matrix. The constructing the data similarity matrix based on the current tag information and the historical tag information includes: Based on the current label information, calculating first similarity information between current sample data in the current multimodal data, and constructing the first pairwise similarity matrix according to the first similarity information; Based on the current label information and the historical label information, second similarity information between the current sample data in the current multimodal data and the historical sample data in the historical multimodal data is calculated, and the second pairwise similarity matrix is ​​constructed according to the second similarity information.

5. The method according to claim 1, characterized in that The performing hash code learning based on the concept vector matrix and the data similarity matrix to obtain a target hash code, and storing the target hash code in a database includes: Constructing a target optimization function based on the concept vector matrix and the data similarity matrix; Initialize the current hash code; Alternately optimizing the current hash code by using the target optimization function to obtain the target hash code; The target hash code is stored in the database.

6. The method according to claim 1, characterized in that The learning of the hash function based on the target hash code to obtain the current hash function, and dynamically updating the historical hash function according to the current hash function to obtain the target hash function, includes: Based on the target hash code, the current multimodal data and the historical multimodal data, a hash function is learned by using a linear regression method to obtain the current hash function; The historical hash function is dynamically updated according to the current hash function to obtain the target hash function.

7. The method according to claim 1, characterized in that The step of calculating the similarity between the hash code to be queried and the target hash code in the database to obtain a target search result includes: Calculating the Hamming distance between the to-be-queried hash code and the target hash code in the database to obtain a similarity calculation result; The target search result is obtained according to the similarity calculation result and the preset search rule.

8. A dynamic cross-modal hash retrieval system based on concept vector learning, characterized in that: The system includes the following modules: A data acquisition module is used to acquire current label information of current multimodal data and historical label information of historical multimodal data; A concept vector matrix construction module is used to construct a concept vector matrix based on the current label information and the Hadamard matrix; wherein the concept vectors in the concept vector matrix are used to characterize that samples with the same label have the same and unchanged vectors; A similarity matrix construction module, used to construct a data similarity matrix based on the current tag information and the historical tag information; A hash code learning module, used for performing hash code learning based on the concept vector matrix and the data similarity matrix to obtain a target hash code, and storing the target hash code in a database; A hash function learning module, configured to obtain a current hash function based on the target hash code, and dynamically update a historical hash function according to the current hash function to obtain a target hash function; A hash code analysis module, used to analyze the data to be queried according to the target hash function to obtain a hash code to be queried corresponding to the data to be queried; The hash search module is used to calculate the similarity between the hash code to be queried and the target hash code in the database to obtain a target search result.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images