Dynamic cross-modal hashing retrieval method and system based on concept vector learning

Abstract

The application discloses a dynamic cross-modal hash retrieval method and system based on concept vector learning, and the method comprises the following steps: constructing a concept vector matrix based on current label information and a Hadamard matrix; the concept vectors in the concept vector matrix are used for representing that samples with the same label have the same and invariable vectors; constructing a data similarity matrix based on current label information and historical label information; obtaining target hash codes based on the concept vector matrix and the data similarity matrix; obtaining a target hash function based on the target hash codes; analyzing the to-be-queried data according to the target hash function to obtain to-be-queried hash codes corresponding to the to-be-queried data; and performing similarity calculation on the to-be-queried hash codes and the target hash codes in a database to obtain target retrieval results. The application can overcome the defects of the concept drift problem in a dynamic data environment, improve the adaptability of a cross-modal hash model to the concept drift, improve the dynamic cross-modal hash retrieval performance, and can be widely applied to the technical field of information processing.

Description

Technical Field

[0001] The present application relates to the field of information processing technology, and in particular to a dynamic cross-modal hash retrieval method and system based on concept vector learning. Background Art

[0002] In the field of cross-modal hashing, current cross-modal hashing is difficult to adapt to the concept drift problem in dynamic data environments, and the concept drift problem in dynamic data environments will lead to a decline in the performance of cross-modal hashing.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0004] The embodiments of the present application aim to solve, at least to a certain extent, one of the technical problems in the related art. To this end, the main purpose of the embodiments of the present application is to propose a dynamic cross-modal hash retrieval method and system based on concept vector learning, which can overcome the defects of the concept drift problem in a dynamic data environment, enhance the adaptability of the cross-modal hash model to concept drift, and improve the performance of dynamic cross-modal hash retrieval.

[0005] To achieve the above objectives, an embodiment of the present application proposes a dynamic cross-modal hash retrieval method based on concept vector learning, which includes the following steps:

[0006] Obtain current label information of current multimodal data and historical label information of historical multimodal data;

[0007] Based on the current label information and the Hadamard matrix, a concept vector matrix is ​​constructed; wherein the concept vectors in the concept vector matrix are used to represent that samples with the same label have the same and unchanged vectors;

[0008] Constructing a data similarity matrix based on the current tag information and the historical tag information;

[0009] Performing hash code learning based on the concept vector matrix and the data similarity matrix to obtain a target hash code, and storing the target hash code in a database;

[0010] Performing hash function learning based on the target hash code to obtain a current hash function, and dynamically updating a historical hash function according to the current hash function to obtain a target hash function;

[0011] Analyze the data to be queried according to the target hash function to obtain a hash code corresponding to the data to be queried;

[0012] The similarity between the hash code to be queried and the target hash code in the database is calculated to obtain a target retrieval result.

[0013] In some embodiments, constructing a concept vector matrix based on the current label information and the Hadamard matrix includes:

[0014] Based on the current label information, assigning a column vector of the Hadamard matrix as a prototype code to a number of current category data in the current multimodal data;

[0015] Based on the current sample label information corresponding to the current sample data in each of the current category data, generating a concept vector corresponding to each of the current sample label information in the concept space;

[0016] The concept vector matrix is ​​constructed based on each of the concept vectors.

[0017] In some embodiments, generating a concept vector corresponding to each current sample label information in the concept space based on the current sample label information corresponding to each current sample data in the current category data includes:

[0018] Based on the current sample label information corresponding to the current sample data in each current category data, the concept vector corresponding to each current sample label information is generated in the concept space by a concept vector calculation formula; the concept vector calculation formula is:

[0019]

[0020] Among them, F i Represented as a concept vector, c i It is expressed as the number of prototype codes corresponding to the i-th sample data, It represents the jth prototype code of the i-th sample data, i represents the sequence number of the sample data, and j represents the sequence number of the prototype code.

[0021] In some embodiments, the data similarity matrix includes a first pairwise similarity matrix and a second pairwise similarity matrix, and constructing the data similarity matrix based on the current tag information and the historical tag information includes:

[0022] Calculating first similarity information between current sample data in the current multimodal data based on the current label information, and constructing a first pairwise similarity matrix according to the first similarity information;

[0023] Based on the current label information and the historical label information, second similarity information between the current sample data in the current multimodal data and the historical sample data in the historical multimodal data is calculated, and the second pairwise similarity matrix is ​​constructed according to the second similarity information.

[0024] In some embodiments, performing hash code learning based on the concept vector matrix and the data similarity matrix to obtain a target hash code, and storing the target hash code in a database includes:

[0025] Constructing a target optimization function based on the concept vector matrix and the data similarity matrix;

[0026] Initialize the current hash code;

[0027] Alternately optimizing the current hash code using the target optimization function to obtain the target hash code;

[0028] The target hash code is stored in the database.

[0029] In some embodiments, performing hash function learning based on the target hash code to obtain a current hash function, and dynamically updating a historical hash function based on the current hash function to obtain a target hash function, includes:

[0030] Based on the target hash code, the current multimodal data, and the historical multimodal data, a linear regression method is used to perform hash function learning to obtain the current hash function;

[0031] The historical hash function is dynamically updated according to the current hash function to obtain the target hash function.

[0032] In some embodiments, calculating the similarity between the query hash code and the target hash code in the database to obtain the target search result includes:

[0033] Calculating the Hamming distance between the hash code to be queried and the target hash code in the database to obtain a similarity calculation result;

[0034] The target search result is obtained according to the similarity calculation result and the preset search rules.

[0035] To achieve the above objectives, another aspect of the present application provides a dynamic cross-modal hash retrieval system based on concept vector learning, which includes the following modules:

[0036] A data acquisition module is used to obtain current label information of current multimodal data and historical label information of historical multimodal data;

[0037] A concept vector matrix construction module is used to construct a concept vector matrix based on the current label information and the Hadamard matrix; wherein the concept vectors in the concept vector matrix are used to represent that samples with the same label have the same and unchanged vectors;

[0038] A similarity matrix construction module, configured to construct a data similarity matrix based on the current tag information and the historical tag information;

[0039] a hash code learning module, configured to perform hash code learning based on the concept vector matrix and the data similarity matrix to obtain a target hash code, and store the target hash code in a database;

[0040] A hash function learning module is configured to obtain a current hash function based on the target hash code, and dynamically update a historical hash function according to the current hash function to obtain a target hash function;

[0041] A hash code analysis module, configured to analyze the data to be queried according to the target hash function to obtain a hash code corresponding to the data to be queried;

[0042] The hash retrieval module is used to calculate the similarity between the hash code to be queried and the target hash code in the database to obtain a target retrieval result.

[0043] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned method when executing the computer program.

[0044] To achieve the above objectives, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0045] The embodiments of the present application include at least the following beneficial effects: the present application provides a dynamic cross-modal hash retrieval method and system based on concept vector learning, which obtains current label information of current multimodal data and historical label information of historical multimodal data; constructs a concept vector matrix based on the current label information and the Hadamard matrix; wherein the concept vectors in the concept vector matrix are used to characterize that samples with the same label have the same and unchanged vectors; constructs a data similarity matrix based on the current label information and the historical label information; performs hash code learning based on the concept vector matrix and the data similarity matrix to obtain a target hash code, and stores the target hash code in a database; performs hash function learning based on the target hash code to obtain a current hash function, and dynamically updates the historical hash function based on the current hash function to obtain a target hash function; analyzes the data to be queried based on the target hash function to obtain a hash code to be queried corresponding to the data to be queried; performs similarity calculation between the hash code to be queried and the target hash code in the database to obtain a target retrieval result. The embodiment of the present application defines the same and unchanging concept vector for samples with the same label in the constructed concept space. The concept vector will not change with the occurrence of concept drift, and then learns the information of the concept vector when learning the hash code, so that the model has the ability to cope with concept drift, overcomes the defect that related cross-modal hashing cannot cope with the concept drift problem in a dynamic data environment, improves the adaptability of the cross-modal hashing model to concept drift, and can still efficiently learn hash functions and hash codes in a dynamic data environment where concept drift frequently occurs, thereby improving the dynamic cross-modal hash retrieval performance; at the same time, it also effectively learns the pairwise similarity matrix corresponding to the current multimodal data and the historical multimodal data. The hash code obtained based on the pairwise similarity matrix and the concept vector matrix can retain the similarity information between a large number of samples, so that the hash code has greater distinguishability. After obtaining the hash code, the hash function can be effectively and accurately updated using the new and old data information and the hash code, thereby improving the efficiency and accuracy of dynamic cross-modal hash retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a flowchart of the steps of the dynamic cross-modal hash retrieval method based on concept vector learning provided by an embodiment of the present application;

[0047] Figure 2 Schematic diagram of the framework of concept vector learning provided by the embodiment of the present application;

[0048] Figure 3 Schematic diagram of a flow chart of a dynamic cross-modal hash retrieval method based on concept vector learning provided in an embodiment of the present application;

[0049] Figure 4 This is a schematic diagram of the framework of online concept vector hashing provided by an embodiment of the present application;

[0050] Figure 5 Schematic diagram of the structure of a dynamic cross-modal hash retrieval system based on concept vector learning provided by an embodiment of the present application;

[0051] Figure 6 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are merely examples of systems and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0053] It will be understood that the terms "first", "second", etc. used in this application may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0054] The terms "at least one", "plurality", "each", "any", etc. used in this application include "at least one", "two" or more, "plurality" or "each", "any" or "any one", "each" or "any one" as used herein.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0056] In the field of cross-modal hashing, current cross-modal hashing is difficult to adapt to the concept drift problem in dynamic data environments, and the concept drift problem in dynamic data environments will lead to a decline in the performance of cross-modal hashing.

[0057] In view of this, an embodiment of the present application provides a dynamic cross-modal hash retrieval method and system based on concept vector learning. The scheme obtains current label information of current multimodal data and historical label information of historical multimodal data; constructs a concept vector matrix based on the current label information and the Hadamard matrix; wherein the concept vectors in the concept vector matrix are used to characterize that samples with the same label have the same and unchanged vectors; constructs a data similarity matrix based on the current label information and the historical label information; performs hash code learning based on the concept vector matrix and the data similarity matrix to obtain a target hash code, and stores the target hash code in a database; performs hash function learning based on the target hash code to obtain a current hash function, and dynamically updates the historical hash function according to the current hash function to obtain a target hash function; analyzes the data to be queried according to the target hash function to obtain a hash code to be queried corresponding to the data to be queried; performs similarity calculation between the hash code to be queried and the target hash code in the database to obtain a target retrieval result. The embodiment of the present application defines the same and unchanging concept vector for samples with the same label in the constructed concept space. The concept vector will not change with the occurrence of concept drift, and then learns the information of the concept vector when learning the hash code, so that the model has the ability to cope with concept drift, overcomes the defect that related cross-modal hashing cannot cope with the concept drift problem in a dynamic data environment, improves the adaptability of the cross-modal hashing model to concept drift, and can still efficiently learn hash functions and hash codes in a dynamic data environment where concept drift frequently occurs, thereby improving the dynamic cross-modal hash retrieval performance; at the same time, it also effectively learns the pairwise similarity matrix corresponding to the current multimodal data and the historical multimodal data. The hash code obtained based on the pairwise similarity matrix and the concept vector matrix can retain the similarity information between a large number of samples, so that the hash code has greater distinguishability. After obtaining the hash code, the hash function can be effectively and accurately updated using the new and old data information and the hash code, thereby improving the efficiency and accuracy of dynamic cross-modal hash retrieval.

[0058] The embodiment of the present application provides a dynamic cross-modal hash retrieval method based on concept vector learning, which relates to the field of information processing technology. The embodiment of the present application provides a dynamic cross-modal hash retrieval method based on concept vector learning, which can be applied to a terminal, a server, or a software running on a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or as a server cluster or distributed system consisting of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network, content distribution network) and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements a dynamic cross-modal hash retrieval method based on concept vector learning, etc., but is not limited to the above forms.

[0059] The present application can be used in numerous general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs (Personal Computers, personal computers), minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0060] See also Figure 1 , Figure 1 This is an optional step flow chart of the dynamic cross-modal hash retrieval method based on concept vector learning provided by an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S107.

[0061] Step S101, obtaining current label information of current multimodal data and historical label information of historical multimodal data;

[0062] Current multimodal data can be referred to as new data or newly arrived data. Current multimodal data refers to a dataset containing multiple types of information collected at the current specific point in time. Historical multimodal data, also known as old data or arrived data, is a multimodal dataset collected at a certain point in time or time period in the past. Compared to current multimodal data, historical multimodal data reflects past states or events and is therefore of great significance for the analysis and processing of subsequent data.

[0063] Multimodal data typically includes data in multiple forms, such as text, images, audio, and video, which together describe multiple aspects of an entity or event. For example, on social media, a post may contain a text description (text), an accompanying image (image), and possibly a video or audio clip, which together constitute multimodal data.

[0064] Current tag information refers to the tags or annotations associated with the current multimodal data, used to describe or categorize the content and features of the current multimodal data. Historical tag information refers to the tags or annotations associated with historical multimodal data, similarly used to describe or categorize the content and features of historical multimodal data.

[0065] In step S101 of the embodiment of the present application, current label information of current multimodal data and historical label information of historical multimodal data are obtained to provide data support for subsequent data processing and analysis.

[0066] Step S102: constructing a concept vector matrix based on the current label information and the Hadamard matrix; wherein the concept vectors in the concept vector matrix are used to represent that samples with the same label have the same and unchanged vectors;

[0067] The concept vector matrix is ​​composed of the concept vectors corresponding to all samples in the current multimodal data. Concept vectors are used to represent that samples with the same label have the same and unchanging vectors. The space where the concept vectors reside is called the concept space. It is easy to understand that regardless of whether concept drift occurs when the multimodal data arrives, samples with the same label have the same and unchanging real-valued concept vector. Since samples with the same label share the same concept vector in the concept space, and this concept vector does not change with the occurrence of concept drift, learning the information in the concept vector can enable the system to have a certain ability to cope with concept drift, overcoming the defect of related cross-modal hash retrieval that cannot cope with the concept drift problem in dynamic data environments, improving the adaptability of cross-modal hashing systems to concept drift, and making it possible to efficiently learn hash functions and hash codes in dynamic data environments where concept drift frequently occurs, and achieve excellent retrieval performance.

[0068] It is worth mentioning that in order to solve the problem of concept drift in the field of cross-modal hashing, the embodiment of the present application proposes online concept vector hashing (OCVH), that is, the dynamic cross-modal hash retrieval method based on concept vector learning provided in the embodiment of the present application. Concept vector learning can better deal with the concept drift problem faced by cross-modal hashing.

[0069] In some embodiments, step S102 may include: based on the current label information, assigning the column vector of the Hadamard matrix as a prototype code to several current category data in the current multimodal data; based on the current sample label information corresponding to the current sample data in each current category data, generating a concept vector corresponding to each current sample label information in the concept space; and constructing a concept vector matrix based on each concept vector.

[0070] Among them, the Hadamard matrix is ​​a special square matrix whose elements are composed of +1 and -1, and the sum of the products of the corresponding elements of any two different rows (or columns) is 0. In an embodiment of the present application, a column of the Hadamard matrix is ​​assigned to each category in the current multimodal data as its prototype code. Since the first column of the Hadamard matrix is ​​an all-1 vector, the allocation starts from the second column of the Hadamard matrix. For example, assuming that there are currently three categories C1, C2, and C3, since the first column of the Hadamard matrix is ​​an all-1 vector, the allocation starts from the second column, then C1 corresponds to the second column, C2 corresponds to the third column, and C3 corresponds to the fourth column.

[0071] The prototype code is a column vector selected from the Hadamard matrix, which is a benchmark code used to represent different categories or labels. By assigning a prototype code to each category, samples can be quickly encoded and classified in the hash space.

[0072] Among them, the current category data refers to the various categories in the current multimodal data. Specifically, the category refers to the different types in the data set. For example, the category can be different object types, such as "cat", "dog", "bird", etc.; in the specific implementation, it is necessary to assign a column of the Hadamard matrix to each category in the current multimodal data as its prototype code. Among them, the current sample data is each sample in the current multimodal data set. The current sample data contains information from different modalities (such as text, images, etc.), and each sample has a corresponding label information, which is called the current sample label information. Specifically, the label is a way to identify the category. The label is usually the name or code of the category to which the instance in the data set belongs. It is easy to understand that in a multi-label or multi-classification problem, a sample can belong to multiple categories, and each category corresponds to a label. For example, a picture containing a cat and a dog may have two labels, such as "cat" and "dog".

[0073] In some embodiments, based on the current sample label information corresponding to the current sample data in each current category data, generating a concept vector corresponding to each current sample label information in the concept space includes: based on the current sample label information corresponding to the current sample data in each current category data, generating a concept vector corresponding to each current sample label information in the concept space using a concept vector calculation formula; the concept vector calculation formula is:

[0074]

[0075] Among them, F i Represented as a concept vector, c i It is expressed as the number of prototype codes corresponding to the i-th sample data, It represents the jth prototype code of the i-th sample data, i represents the sequence number of the sample data, and j represents the sequence number of the prototype code.

[0076] In the specific implementation, the process of concept vector learning is:

[0077] First, the OCVH provided in the embodiment of the present application assigns a column of the Hadamard matrix as its prototype code for each category of the current multimodal data (since the first column of the Hadamard matrix is ​​an all-1 vector, the allocation starts from the second column). The calculation formula of the Hadamard matrix prototype code is as follows:

[0078] G i =H i+1

[0079] Where H∈{-1, 1} r×r Represents the Hadamard matrix; i∈{1,2,3......c t}, i represents the serial number of the category, c trepresents the number of categories at time t; G i Represents the prototype code of the i-th category; H i+1 represents the i+1th column of the Hadamard matrix.

[0080] It should be noted that all i and j in the formula in the embodiments of the present application represent the position order in the corresponding context. If i is 1, 2, or 3, then it is the 1st, 2nd, and 3rd.

[0081] Secondly, the current multimodal data X m The concept vector F of the i-th sample in i The concept vector calculation formula is:

[0082]

[0083] Among them, F i Represented as a concept vector; c i It is expressed as the number of prototype codes corresponding to the i-th sample data, that is, the number of labels of the i-th sample; It represents the jth prototype code of the i-th sample data; i represents the sequence number of the sample data, i∈{1,2,3......n}; j represents the sequence number of the prototype code.

[0084] Optionally, the concept vector matrix F r×n is the current multimodal data X m The concept vectors corresponding to all samples in , where r represents the length of the hash code and n represents the current multimodal data X m According to the above concept vector calculation formula, regardless of whether the multimodal data arrives in time or not, samples with the same label have the same and unchanged real-valued concept vector.

[0085] Furthermore, to cope with concept drift, when learning hash codes, the hash code of a sample should be close to the concept vectors of similar samples (including itself) and away from the concept vectors of dissimilar samples. The concept vector optimization formula for coping with concept drift to maintain concept vector similarity is as follows:

[0086]

[0087] Where B∈{-1, 1} r×n Represents the current multimodal data (newly arrived data) X m The corresponding hash code, it should be noted that B is composed of all current multimodal data X m The matrix composed of the corresponding hash codes, B is an r×n matrix, r represents the length of the hash code, and n represents the current multimodal data X mThe number of samples; F represents the concept vector matrix; S represents the pairwise similarity matrix between each sample in the current multimodal data, that is, the similarity matrix between new data, hereinafter referred to as the first pairwise similarity matrix; T represents the matrix transpose.

[0088] It should be noted that the above concept vector optimization formula means saving the information in the concept vectors of all samples in the current multimodal data in the hash code B. The concept vector optimization formula can also be called a concept vector learning item.

[0089] The OCVH method provided in the embodiment of the present application predefines an identical and unchanging vector for samples with the same label. This vector is called a concept vector, and the space where the concept vector is located is called a concept space. In the embodiment of the present application, when learning the hash code, the OCVH method can make the hash code of the sample close to the area where the concept vector of its similar samples is located in the concept space, and away from the area where the concept vector of its dissimilar samples is located. Since samples with the same label share an identical concept vector in the concept space, and this vector does not change with the occurrence of concept drift, the OCVH method provided in the embodiment of the present application has a certain ability to cope with concept drift by learning the information in the concept vector.

[0090] Specifically, see Figure 2 , Figure 2 : is a schematic diagram of the framework of concept vector learning provided by the embodiment of the present application; Figure 2 As shown in the figure, before the concept drift occurs, it is assumed that there are two classes in the image modality and the text modality, namely C1 and C2; after the concept drift occurs, the data distribution within the C1 category and the C2 category changes, and a new category C3 also appears. After the concept drift, the hash codes corresponding to the samples with only C1 and C2 labels are still projected to their original corresponding areas in the concept space, while the hash codes corresponding to the samples with C3 labels will be projected to their corresponding new areas. The impact of concept drift has been offset to a certain extent.

[0091] In step S102 shown in the embodiment of the present application, by defining that samples with the same label share the same concept vector in the concept space, and the concept vector will not change with the occurrence of concept drift, the system can have a certain ability to cope with concept drift by learning the information in the concept vector, thereby overcoming the defect that the related cross-modal hashing cannot cope with the concept drift problem in a dynamic data environment, and improving the adaptability of the cross-modal hashing system to concept drift, so that in a dynamic data environment where concept drift frequently occurs, hash functions and hash codes can still be efficiently learned, and excellent retrieval performance can be achieved.

[0092] Step S103: constructing a data similarity matrix based on the current tag information and the historical tag information;

[0093] The data similarity matrix is ​​used to characterize the similarity between samples. The data similarity matrix is ​​calculated based on the label information and includes a first pairwise similarity matrix and a second pairwise similarity matrix. The first pairwise similarity matrix can be represented by S. The first pairwise similarity matrix is ​​the current multimodal data X m The pairwise similarity matrix between each sample in , that is, the similarity matrix between new data; the second pairwise similarity matrix can be used Indicates that the second pairwise similarity matrix is ​​the current multimodal data X m and historical multimodal data The pairwise similarity matrix between each sample in , that is, the similarity matrix between new data and old data.

[0094] In some embodiments, step S103 may include: calculating first similarity information between current sample data in the current multimodal data based on current label information, and constructing a first pairwise similarity matrix based on the first similarity information; calculating second similarity information between current sample data in the current multimodal data and historical sample data in the historical multimodal data based on current label information and historical label information, and constructing a second pairwise similarity matrix based on the second similarity information.

[0095] The first similarity information is the similarity between samples in the current multimodal data, which is calculated based on the label information corresponding to the samples in the current multimodal data and is used to construct the first pairwise similarity matrix. The second similarity information is the similarity between samples in the current multimodal data and samples in the historical multimodal data, which is calculated based on the label information corresponding to the samples in the current multimodal data and the samples in the historical multimodal data and is used to construct the second pairwise similarity matrix.

[0096] In the specific implementation, the specific implementation process of similarity learning between new data (similarity learning between current multimodal data) and similarity learning between new and old data (similarity learning between current multimodal data and historical multimodal data) is as follows:

[0097] (1) Similarity learning between new data:

[0098] The goal of hash code learning is to preserve the similarity between samples. Since samples of different modalities share the same label space, labels are used to calculate the similarity between samples. The initial calculation formula for the pairwise similarity matrix of new data samples is as follows:

[0099]

[0100] Among them, S ijRepresents the pairwise similarity matrix of the new data sample (not fine-grained optimization); L i represents the i-th column of the label matrix; L j The jth column of the label matrix; Represents the inner product operator.

[0101] However, since the initial calculation formula of the pairwise similarity matrix of the new data sample cannot meet the fine-grained requirements, in the embodiment of the present application, a fine-grained pairwise similarity matrix calculation formula is used to calculate the similarity matrix. The fine-grained calculation formula of the pairwise similarity matrix of the new data sample is as follows:

[0102] S=2V T V-11 T

[0103] Where S represents the pairwise similarity matrix of the new data sample (fine-grained optimization), that is, the first pairwise similarity matrix, V represents the 2-norm normalized label matrix corresponding to the new data, and the i-th column of V is defined as V i =L i / ||L i ||, Represents the current multimodal data (newly arrived data) X m The corresponding label matrix, c t represents the number of categories at time t, and n represents the current multimodal data X m The number of samples; V T V calculates the cosine similarity between sample labels and then maps it to [-1, 1] to obtain the pairwise similarity matrix S; T represents the matrix transpose, and 1 represents the all-1 vector.

[0104] Since the Hamming distance between hash codes of similar samples should be as small as possible, and the Hamming distance between hash codes of dissimilar samples should be as large as possible, in order to retain the similarity information in the pairwise similarity matrix B of the new data sample, the following initial new sample similarity retention formula can be used to achieve this:

[0105]

[0106] Where B∈{-1, 1} r×n Represents the current multimodal data (newly arrived data) X m The corresponding hash code, it should be noted that B is composed of all current multimodal data X m The matrix composed of the corresponding hash codes, B is an r×n matrix, r represents the length of the hash code, and n represents the current multimodal data X m The number of samples; B i represents the i-th column of B; B j represents the jth column of B; at time t, the newly arrived data is m represents different modalities and m e {1, 2}, d m represents the dimension of m-th modality feature, n represents the number of samples of newly arrived data; represents a real value space; S ij represents the pair-wise similarity matrix of new data samples (not fine-grained optimization); ||·|| represents the 2-norm of a vector or the Frobenius norm of a matrix.

[0107] Since the above initial new sample similarity preservation formula has a minimization problem, the embodiments of the present application optimize the minimization problem to make the similarity information more easily preserved in the pair-wise similarity matrix B of new data samples. In order to facilitate optimization, the initial new sample similarity preservation formula can be simplified to the following target new sample similarity preservation formula, which can be referred to as a similarity learning term between new data. The target new sample similarity preservation formula is as follows:

[0108]

[0109] Wherein, tr{·} represents the trace function.

[0110] It is easy to understand that the similarity information of the current multi-modal data (newly arrived data) X m can be preserved in the hash code corresponding to the current multi-modal data (newly arrived data) X m by the above target new sample similarity preservation formula.

[0111] (2) Similarity learning between new and old data:

[0112] The hash code of the sample in the current multi-modal data (newly arrived data) X m should be similar to the hash code of its similar sample in the historical multi-modal data (previously arrived data) , and vice versa, the hash code of the sample in the current multi-modal data is far away from the hash code of its dissimilar sample in the historical multi-modal data. Wherein, the calculation formula of the pair-wise similarity matrix between new and old data is as follows:

[0113]

[0114] Wherein, represents the pair-wise similarity matrix between new and old data, i.e. the second pair-wise similarity matrix; represents the 2-norm normalized label matrix corresponding to the old data, the i-th column of which is represents the label matrix corresponding to the historical multi-modal data (previously arrived data) , N represents the number of samples of previously arrived data, c trepresents the number of categories at time t, in particular, The number of categories in is the same as the number of categories at time t. If a new category appears at time t, a full 0 matrix is ​​used to replace The dimension is extended to c t ×N; T represents matrix transpose; where, m represents different modes and m∈{1,2},d m represents the dimension of m-modal features, represents a real-valued space.

[0115] Furthermore, in order to preserve the similarity information between the new and old data in the pairwise similarity matrix B of the new data sample, this can be achieved by the following target new and old sample similarity preservation formula, specifically:

[0116]

[0117] in, Representation and historical multimodal data (data that has been reached before) The corresponding hash code, N represents the number of samples of data that have arrived before, and r represents the length of the hash code.

[0118] It's worth noting that subsequent hash code learning requires three types of information: similarity between new data, information in the concept vector, and similarity between new and old data. New data corresponds to its hash code, and old data corresponds to its hash code. At time t, only the hash code B of the new data arriving at time t needs to be learned; the hash code of the old data has already been learned at the previous time.

[0119] In step S103 shown in the embodiment of the present application, by constructing a first pairwise similarity matrix and a second pairwise similarity matrix, the similarity between samples in the current multimodal data and the similarity between samples in the current multimodal data and the historical multimodal data are comprehensively characterized. The construction of the similarity matrix provides an important data basis for subsequent hash code learning and hash function learning; and, through the similarity preservation formula, the similarity information between the new data and the similarity information between the new and old data are retained in the hash code corresponding to the current multimodal data, providing a data basis for subsequent hash code learning.

[0120] Step S104: performing hash code learning based on the concept vector matrix and the data similarity matrix to obtain a target hash code, and storing the target hash code in a database;

[0121] Among them, the concept vector matrix contains the information in the concept vector, and the data similarity matrix contains the similarity information between new data and the similarity information between new and old data.

[0122] In some embodiments, step S104 may include: constructing a target optimization function based on the concept vector matrix and the data similarity matrix; initializing a current hash code; alternately optimizing the current hash code using the target optimization function to obtain a target hash code; and storing the target hash code in a database.

[0123] The acquisition process of the target optimization function is as follows:

[0124] In the embodiment of the present application, in order to make the hash code contain as much information as possible, it is necessary to maximize the variance of the hash code to obtain higher entropy, which is specifically achieved by the following hash code variance maximization formula:

[0125]

[0126] In addition, the hash code B of the new data needs to be subject to bit balance constraints and bit irrelevant constraints to obtain a higher quality hash code. The overall objective function expression of hash code learning is as follows:

[0127]

[0128] stB1=0,BB T =nE

[0129] Among them, α, β, and γ are all trade-off parameters, B1=0 is used to ensure that the hash code B of the new data is balanced, BB T =nE is used to ensure that the hash code B of the new data is unrelated, and n represents the number of samples corresponding to the newly arrived data. It should be noted that n in the embodiments of the present application represents the number of samples corresponding to the newly arrived data, r represents the hash code length, E represents the identity matrix, and 0 represents the all-zero matrix.

[0130] However, since the hash code B of the new data is a discrete value matrix, when these two constraints are imposed on the hash code B of the new data, the overall objective function expression becomes an NP-Hard (Non-deterministic Polynomial-time Hard) problem. Therefore, in the embodiment of the present application, the B on the right side of each term in the overall objective function expression is replaced by a real-valued matrix To deal with this situation, the constraint on B is relaxed to Y. The final objective optimization function expression of hash code learning is as follows:

[0131]

[0132] st Y1=0,YY T =nE

[0133] It should be noted that in the process of hash code learning, at time t, only the hash code B of the new data arriving at time t needs to be learned; the hash code of the old data The learning has been done at the previous moment, and there is no need to learn the hash code again.

[0134] In the embodiment of the present application, the hash code B of the current new data (ie, the current hash code) is first initialized, and then the final hash code B of the new data (ie, the target hash code) and the real-valued matrix Y are solved using an alternating optimization process.

[0135] The current hash code is initialized using the hash code initialization formula. Since the current hash code does not meet the quality requirements of the target optimization function, it is necessary to perform alternating optimization on the current hash code to obtain the final target hash code B. The specific process of initializing the current hash code is as follows:

[0136] First, perform eigendecomposition on the pairwise similarity matrix S of the new data sample. The eigendecomposition formula is as follows:

[0137]

[0138] Among them, λ i represents the i-th largest eigenvalue; J i represents the normalized eigenvector corresponding to the i-th largest eigenvalue; n represents the number of samples of newly arrived data; i represents the bit order. For example, assuming there are three eigenvalues, namely 1, 2, and 3, the first largest eigenvalue is 3, the second largest eigenvalue is 2, and the third largest eigenvalue is 1.

[0139] Next, initialize the current hash code using the following hash code initialization formula:

[0140] B=sgn([J2,J3,......J r+1 ] T )

[0141] Among them, sgn represents an element-by-element operator, which is 1 when the element is greater than 0, otherwise it is -1.

[0142] Optionally, alternating optimization is to optimize the hash code by repeated iterations until a satisfactory solution of the target optimization function is reached to obtain the target hash code. In a specific implementation, the process of performing alternating optimization based on the target optimization function to solve the final hash code B of the new data is to alternately optimize the hash code B of the current new data and the real-valued matrix Y to obtain the final hash code B (i.e., the target hash code) and the real-valued matrix Y of the new data:

[0143] 1) Update the real-valued matrix Y

[0144] First, the hash code B of the new data is fixed when updating the real-valued matrix Y. According to the expression of the above objective optimization function, the following real-valued matrix updating expression can be obtained, specifically:

[0145]

[0146] s.t. Y1=0,YY T =nE

[0147] Next, the calculation formula of the pair-wise similarity matrix between the new and old data is substituted into the above real-valued matrix updating expression, and the real-valued matrix updating expression is rewritten in the following form, which can be called the target real-valued matrix updating expression:

[0148]

[0149] s.t. Y1=0,YY T =nE

[0150] wherein an auxiliary variable M is defined, and specifically, the expression of the auxiliary variable M is as follows:

[0151]

[0152] Optionally, the historical multi-modal data (data previously reached) corresponding hash code and the corresponding 2-norm normalized label matrix are rewritten as and and respectively, wherein B t-1 and V t-1 respectively represent the hash code and the corresponding 2-norm normalized label matrix of the data arrived at the t-1 moment. Then the following first intermediate matrix recursive formula can be obtained:

[0153]

[0154] wherein the intermediate matrix K1 is defined as and the intermediate matrix K2 is defined as The above first intermediate matrix recursive formula can be calculated by accessing the intermediate matrix and the intermediate matrix and the new data to calculate the auxiliary variable M, without directly accessing the old data with a size of N.

[0155] Specifically, the centering matrix is defined, and the intermediate value matrix MDM generated in the solving process is​T Perform SVD decomposition on it, and the SVD decomposition formula is as follows:

[0156]

[0157] in, represents a positive diagonal matrix, Represents the median matrix MDM T rank, represents the corresponding eigenvector of a positive eigenvalue, Represents the corresponding eigenvector of the remaining eigenvalues. Therefore, the closed-form solution of the above target real-valued matrix update expression can be calculated by the following closed-form solution formula of the target real-valued matrix:

[0158]

[0159] in, Indicates that through The orthogonal matrix obtained by executing the Gram-Schmidt process, Ψ (no special meaning, an intermediate value generated during the solution process) is calculated as Ψ = DM T IΛ -1 / 2 , represents a random orthogonal matrix. hour, and Empty.

[0160] 2) Update the hash code B of the new data:

[0161] When updating the hash code B of the new data, the real-valued matrix Y is fixed, and the real-valued matrix Y at this time is the updated matrix. Therefore, the expression of the above target optimization function can be simplified to the following formula. The simplified expression of the target optimization function can be called the target solution formula for the hash code of the new data, specifically:

[0162]

[0163] Then the target solution formula based on the hash code of the new data can directly solve the hash code B of the new data. The expression of the hash code B of the solved new data is as follows:

[0164] B=sgn(SY T +γY T )

[0165] At this time, the hash code B of the new data is the updated hash code.

[0166] In step S104 shown in the embodiment of the present application, a set of optimized hash codes (i.e., target hash codes) are generated by performing hash code learning using the concept vector matrix and the data similarity matrix. This process combines the hash code constraints of the target optimization function and the alternating optimization strategy to ensure that the generated hash codes can accurately reflect the similarity or semantic relationship between the data.

[0167] Step S105: performing hash function learning based on the target hash code to obtain a current hash function, and dynamically updating a historical hash function according to the current hash function to obtain a target hash function;

[0168] In some embodiments, step S105 may include: performing hash function learning using a linear regression method based on the target hash code, current multimodal data, and historical multimodal data to obtain a current hash function; and dynamically updating the historical hash function according to the current hash function to obtain a target hash function.

[0169] For hash function learning, it is a mapping function obtained through training data (including target hash code, current multimodal data and historical multimodal data), which can map the input data sample to a corresponding binary hash code representation. Among them, the mapping process should try to preserve the similarity relationship between data samples, that is, the hash codes of similar samples are close in Hamming distance, while the hash codes of dissimilar samples are far apart. In the embodiment of the present application, the linear regression method is used for hash function learning. For the specific implementation principle of the linear regression method, those skilled in the art can refer to the implementation scheme of the relevant technology, and the embodiment of the present application will not be described in detail here.

[0170] The current hash function refers to the hash function learned based on the target hash code, current multimodal data, and historical multimodal data. The current hash function can reflect the distribution and characteristics of the current data and can map new data samples to the corresponding hash code. However, as new data is continuously added, the data distribution may change, so the current hash function needs to update the historical hash function to adapt to this change. The historical hash function refers to the previously learned hash function, which is learned based on the data set and hash code that arrived at the previous time.

[0171] The target hash function is obtained by dynamically updating the historical hash function with the current hash function, and is the latest target hash function.

[0172] In the specific implementation, the specific implementation process of hash function learning is as follows:

[0173] After learning the hash code, the OCVH method provided in the embodiment of the present application needs to learn the hash function through the hash code. Specifically, the newly arrived data X m and the newly arrived data Xm The corresponding hash code B is learned through the linear regression method. P m Representing the hash function of m-mode, the initial expression of the linear regression learning hash function is as follows:

[0174]

[0175] Where μ represents a trade-off parameter. However, the initial expression of the linear regression learning hash function only considers new data, not old data. As new data is continuously added, the model's retrieval performance for old data will inevitably decline. To address this situation, the embodiment of the present application also considers old data when learning the hash function. The target expression of the optimized linear regression learning hash function is as follows:

[0176]

[0177] Among them, let the above formula (the target expression of linear regression learning hash function) be the hash function P m The partial derivative of is zero, then the hash function P m It can be solved by the following hash function calculation formula, specifically:

[0178]

[0179] Among them, historical multimodal data (data that has arrived before) was rewritten as represents the data that arrived before time t-1, X m t-1 represents the data arriving at time t-1. Then we can get the following recursive formula for the second intermediate matrix:

[0180]

[0181] Among them, the intermediate matrix K3 is defined as The intermediate matrix K4 is According to the above second intermediate matrix recursion formula, the hash function P m No need to access old data of size N, only the intermediate matrix needs to be accessed Intermediate Matrix and new data.

[0182] In step S105 shown in the embodiment of the present application, a current hash function is obtained by learning a hash function using a linear regression method by combining the target hash code, current multimodal data, and historical multimodal data. The historical hash function is then dynamically updated based on the current hash function to obtain a target hash function that is more adaptable to data changes, so that the target hash function can accurately reflect the similarities and differences of the data.

[0183] Step S106, analyzing the data to be queried according to the target hash function to obtain a hash code corresponding to the data to be queried;

[0184] The data to be queried refers to the original data on which the user or system wishes to perform some operation (such as search, comparison, verification, etc.). The data to be queried can be any form of information such as text, image, audio, video, etc.

[0185] The hash code to be queried is the output obtained after the queried data is processed by the target hash function, and is used for similarity matching with the target hash code stored in the database.

[0186] In step S106 of the embodiment of the present application, by analyzing the query data according to the target hash function, a query hash code corresponding to the query data is obtained, which can quickly locate the data items that may match in the database, and then further perform accurate comparison.

[0187] Step S107 : performing similarity calculation between the hash code to be queried and the target hash code in the database to obtain a target search result.

[0188] In some embodiments, step S107 may include: calculating the Hamming distance between the queried hash code and the target hash code in the database to obtain a similarity calculation result; and obtaining a target search result based on the similarity calculation result and a preset search rule.

[0189] Among them, similarity calculation refers to the calculation of the Hamming distance between the query hash code and the target hash code in the database. The Hamming distance between similar sample hash codes is small, and the Hamming distance between dissimilar sample hash codes is large.

[0190] By retrieving samples of other modalities that are close to the query sample in the Hamming space, cross-modal retrieval can be efficiently achieved. For example, taking image query text cross-modal retrieval as an example, cross-modal hashing first uses existing samples in the current database to train the model to obtain a hash function and hash code. The hash codes of image samples and text samples are located in the same Hamming space. When executing a query, the hash function projects the image sample to be queried into this Hamming space and obtains its hash code. It then calculates the Hamming distance between the hash code of the text sample in the database and the hash code of the image sample to be queried, and returns the text sample with the smaller Hamming distance as the retrieval result.

[0191] Among them, the preset search rule can be the number of search results output, or it can be the search results output within a certain Hamming distance threshold. The preset search rule can be set according to actual conditions, and the embodiment of the present application does not limit this.

[0192] In the specific retrieval process, assuming that the query data given by modality m is in, Indicates the number of samples to be queried. The corresponding hash code B q The above hash code learning process can be calculated as Finally with B q Samples with smaller inter-Hamming distances are returned as query results.

[0193] In step S17 shown in the embodiment of the present application, dynamic cross-modal retrieval can be efficiently implemented by searching for samples of other modalities that are close to the query sample in the Hamming space.

[0194] In steps S101 to S107 shown in the embodiment of the present application, current label information of current multimodal data and historical label information of historical multimodal data are obtained; a concept vector matrix is ​​constructed based on the current label information and the Hadamard matrix; wherein the concept vectors in the concept vector matrix are used to characterize that samples with the same label have the same and unchanged vectors; a data similarity matrix is ​​constructed based on the current label information and the historical label information; hash code learning is performed based on the concept vector matrix and the data similarity matrix to obtain a target hash code, and the target hash code is stored in a database; hash function learning is performed based on the target hash code to obtain a current hash function, and the historical hash function is dynamically updated according to the current hash function to obtain a target hash function; the data to be queried is analyzed according to the target hash function to obtain a hash code to be queried corresponding to the data to be queried; similarity is calculated between the hash code to be queried and the target hash code in the database to obtain a target retrieval result. The embodiment of the present application defines the same and unchanging concept vector for samples with the same label in the constructed concept space. The concept vector will not change with the occurrence of concept drift, and then learns the information of the concept vector when learning the hash code, so that the model has the ability to cope with concept drift, overcomes the defect that related cross-modal hashing cannot cope with the concept drift problem in a dynamic data environment, improves the adaptability of the cross-modal hashing model to concept drift, and can still efficiently learn hash functions and hash codes in a dynamic data environment where concept drift frequently occurs, thereby improving the dynamic cross-modal hash retrieval performance; at the same time, it also effectively learns the pairwise similarity matrix corresponding to the current multimodal data and the historical multimodal data. The hash code obtained based on the pairwise similarity matrix and the concept vector matrix can retain the similarity information between a large number of samples, so that the hash code has greater distinguishability. After obtaining the hash code, the hash function can be effectively and accurately updated using the new and old data information and the hash code, thereby improving the efficiency and accuracy of dynamic cross-modal hash retrieval.

[0195] In summary, the OCVH method provided in the embodiments of the present application defines the same and unchanging real-valued concept vector for samples with the same label in the constructed concept space, and learns the information of the concept vector when learning the hash code, so that the dynamic cross-modal hashing has the ability to cope with concept drift; at the same time, the OCVH method effectively learns the pairwise similarity matrix between new data and new data, as well as the pairwise similarity matrix between old data and new data. The hash code obtained based on the pairwise similarity matrix and the concept vector matrix retains the similarity information between samples more completely, so the hash code has good discriminability; after obtaining the hash code, the OCVH method uses the new and old data information and the hash code through linear regression to effectively update the hash function, thereby improving the dynamic cross-modal hash retrieval performance.

[0196] In order to explain the principle of the technical solution of the present invention in detail, the overall process of the present invention is described below in combination with some specific embodiments. It is easy to understand that the following is an explanation of the technical principle of the present invention and cannot be regarded as a limitation of the present invention.

[0197] The algorithm code corresponding to the learning process of the OCVH method provided in the embodiment of the present application at time t can be expressed as follows, wherein the hyperparameters are set as r=64, α=0.5, β=0.1, γ=0.1, μ=1, and the maximum number of iterations is 10. Specifically:

[0198] "Input: Newly arrived data X m and newly arrived data X m The corresponding label matrix L, hash code length r, parameters α, β, γ, μ, auxiliary variables (intermediate matrices) K1, K2, K3, K4 at time t-1;

[0199] 1. Initialize the Hadamard matrix H;

[0200] 2. Expand the dimension of K1 to r×c using an all-zero matrix t ;

[0201] 3. Initialize the hash code B of the new data according to the hash code initialization formula;

[0202] 4. Enter the loop;

[0203] 5. Update the real-valued matrix Y according to the closed-form solution formula of the target real-valued matrix;

[0204] 6. Update the hash code B according to the target solution formula of the hash code of the new data;

[0205] 7. Until convergence or the maximum number of iterations is reached;

[0206] 8. Update the hash function P according to the hash function calculation formula m ;

[0207] 9. Update the auxiliary variables K1, K2, K3, and K4 according to the first intermediate matrix recursion formula and the second intermediate matrix recursion formula;

[0208] Output: Hash code B, hash function P m , updated auxiliary variables K1, K2, K3, K4".

[0209] See also Figure 3 , Figure 3 is a flow chart of a dynamic cross-modal hash retrieval method based on concept vector learning provided by an embodiment of the present application; Figure 3 As shown, Figure 3This is a specific implementation flow chart of the OCVH method proposed in the embodiment of this application to achieve cross-modal retrieval. The overall process of dynamic cross-modal hash retrieval based on concept vector learning is as follows:

[0210] like Figure 3 As shown in the figure, suppose that the top 100 samples of other modalities that are most relevant to the query sample of a certain modality are retrieved from the database. On the one hand, at time t, the image data and text data are first input into the OCVH method. The OCVH method obtains the hash code and updates the hash function of each modality according to the algorithm code corresponding to the learning process at time t. Then, the image data, text data and their corresponding hash codes are stored in the database. For details, please refer to Figure 4 , Figure 4 Schematic diagram of the framework of online concept vector hashing provided by the embodiment of the present application; Figure 4As shown, the learning process of the OCVH method proposed in the embodiment of the present application is mainly divided into hash code learning and hash function learning. Among them, the hash code learning process is divided into two parts: one is concept vector learning, and the other is similarity learning. Among them, concept vector learning is the focus of the embodiment of the present application. Specifically, the model first assigns a column of the Hadamard matrix as its prototype code for each category of the data at time t according to the label of the data at time t (image data at time t and text data at time t), and then generates its corresponding concept vector in the concept space for each sample of the data at time t according to its label. The concept vectors corresponding to all samples of the data at time t constitute a concept vector matrix. The OCVH method uses the concept vector matrix to perform concept vector learning to obtain the ability to cope with concept drift. During the similarity learning process, the OCVH method first calculates the similarity matrix between the new data using the labels of the data at time t (image data at time t and text data at time t), and calculates the similarity matrix between the new and old data using the labels of the data before time t (image data before time t and text data before time t). Then, using the similarity matrix between the new data and the similarity matrix between the new and old data, similarity learning is performed to retain the similarity information between samples in the hash code. After completing the hash code learning, the OCVH method updates the hash function of the text modality and the hash function of the image modality based on the data at all times (time t and before time t) and their corresponding hash codes. On the other hand, the retrieval process is specifically as follows: for the query image, the updated image hash function is used to obtain the query image hash code corresponding to the query image, and then the Hamming distance between the query image hash code and the hash codes of all texts in the database is calculated, and the 100 texts with the smallest Hamming distance to the query image hash code are returned as retrieval results; similarly, for the query text, the updated text hash function is used to obtain the query text hash code corresponding to the query text, and then the Hamming distance between the query text hash code and the hash codes of all images in the database is calculated, and the 100 images with the smallest Hamming distance to the query text hash code are returned as retrieval results.

[0211] It should be pointed out that this embodiment only provides a brief schematic illustration of the general process of the dynamic cross-modal hash retrieval method based on concept vector learning. The detailed description of each step can refer to the relevant content in the aforementioned embodiment and will not be repeated here. It can be understood that the present invention is not limited to this.

[0212] The embodiment of the present application obtains current label information of current multimodal data and historical label information of historical multimodal data; constructs a concept vector matrix based on the current label information and the Hadamard matrix; wherein the concept vectors in the concept vector matrix are used to characterize that samples with the same label have the same and unchanged vectors; constructs a data similarity matrix based on the current label information and the historical label information; performs hash code learning based on the concept vector matrix and the data similarity matrix to obtain a target hash code, and stores the target hash code in a database; performs hash function learning based on the target hash code to obtain a current hash function, and dynamically updates the historical hash function according to the current hash function to obtain a target hash function; analyzes the data to be queried according to the target hash function to obtain a hash code to be queried corresponding to the data to be queried; performs similarity calculation between the hash code to be queried and the target hash code in the database to obtain a target retrieval result. The embodiment of the present application defines the same and unchanging concept vector for samples with the same label in the constructed concept space. The concept vector will not change with the occurrence of concept drift, and then learns the information of the concept vector when learning the hash code, so that the model has the ability to cope with concept drift, overcomes the defect that related cross-modal hashing cannot cope with the concept drift problem in a dynamic data environment, improves the adaptability of the cross-modal hashing model to concept drift, and can still efficiently learn hash functions and hash codes in a dynamic data environment where concept drift frequently occurs, thereby improving the dynamic cross-modal hash retrieval performance; at the same time, it also effectively learns the pairwise similarity matrix corresponding to the current multimodal data and the historical multimodal data. The hash code obtained based on the pairwise similarity matrix and the concept vector matrix can retain the similarity information between a large number of samples, so that the hash code has greater distinguishability. After obtaining the hash code, the hash function can be effectively and accurately updated using the new and old data information and the hash code, thereby improving the efficiency and accuracy of dynamic cross-modal hash retrieval.

[0213] See also Figure 5 The embodiment of the present application further provides a dynamic cross-modal hash retrieval system 500 based on concept vector learning, which can implement the above-mentioned dynamic cross-modal hash retrieval method based on concept vector learning. The system 500 includes the following modules:

[0214] The data acquisition module 501 is used to acquire current label information of current multimodal data and historical label information of historical multimodal data;

[0215] A concept vector matrix construction module 502 is configured to construct a concept vector matrix based on the current label information and the Hadamard matrix; wherein the concept vectors in the concept vector matrix are used to represent that samples with the same label have the same and unchanged vectors;

[0216] A similarity matrix construction module 503 is used to construct a data similarity matrix based on the current tag information and the historical tag information;

[0217] a hash code learning module 504 for performing hash code learning based on the concept vector matrix and the data similarity matrix to obtain a target hash code, and storing the target hash code in a database;

[0218] A hash function learning module 505 is configured to obtain a current hash function based on the target hash code, and dynamically update a historical hash function according to the current hash function to obtain a target hash function;

[0219] A hash code analysis module 506 is configured to analyze the data to be queried according to the target hash function to obtain a hash code corresponding to the data to be queried;

[0220] The hash retrieval module 507 is configured to calculate similarity between the hash code to be queried and the target hash code in the database to obtain a target retrieval result.

[0221] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0222] The present application also provides an electronic device comprising a memory and a processor. The memory stores a computer program, and the processor, when executing the computer program, implements the dynamic cross-modal hash retrieval method based on concept vector learning. The electronic device can be any smart terminal, including a tablet computer and an in-vehicle computer.

[0223] It can be understood that the contents of the above method embodiments are applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0224] See also Figure 6 , Figure 6 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0225] The processor 601 may be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0226] The memory 602 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 602 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 602 and is called by the processor 601 to execute the dynamic cross-modal hash retrieval method based on concept vector learning in the embodiments of this application.

[0227] Input / output interface 603, used to implement information input and output;

[0228] Communication interface 604, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, Wi-Fi, Bluetooth, etc.);

[0229] Bus 605 , which transmits information between various components of the device (e.g., processor 601 , memory 602 , input / output interface 603 , and communication interface 604 );

[0230] The processor 601 , the memory 602 , the input / output interface 603 and the communication interface 604 are connected to each other in communication within the device via a bus 605 .

[0231] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned dynamic cross-modal hash retrieval method based on concept vector learning.

[0232] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiment, the functions specifically implemented by the present storage medium embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0233] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0234] The embodiments of the present application provide a dynamic cross-modal hash retrieval method based on concept vector learning and a dynamic cross-modal hash retrieval system based on concept vector learning, which obtain current label information of current multimodal data and historical label information of historical multimodal data; construct a concept vector matrix based on the current label information and the Hadamard matrix; wherein the concept vectors in the concept vector matrix are used to characterize that samples with the same label have the same and unchanged vectors; construct a data similarity matrix based on the current label information and the historical label information; perform hash code learning based on the concept vector matrix and the data similarity matrix to obtain a target hash code, and store the target hash code in a database; perform hash function learning based on the target hash code to obtain a current hash function, and dynamically update the historical hash function based on the current hash function to obtain a target hash function; analyze the data to be queried based on the target hash function to obtain a hash code to be queried corresponding to the data to be queried; perform similarity calculation between the hash code to be queried and the target hash code in the database to obtain a target retrieval result. The embodiment of the present application defines the same and unchanging concept vector for samples with the same label in the constructed concept space. The concept vector will not change with the occurrence of concept drift, and then learns the information of the concept vector when learning the hash code, so that the model has the ability to cope with concept drift, overcomes the defect that related cross-modal hashing cannot cope with the concept drift problem in a dynamic data environment, improves the adaptability of the cross-modal hashing model to concept drift, and can still efficiently learn hash functions and hash codes in a dynamic data environment where concept drift frequently occurs, thereby improving the dynamic cross-modal hash retrieval performance; at the same time, it also effectively learns the pairwise similarity matrix corresponding to the current multimodal data and the historical multimodal data. The hash code obtained based on the pairwise similarity matrix and the concept vector matrix can retain the similarity information between a large number of samples, so that the hash code has greater distinguishability. After obtaining the hash code, the hash function can be effectively and accurately updated using the new and old data information and the hash code, thereby improving the efficiency and accuracy of dynamic cross-modal hash retrieval.

[0235] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0236] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0237] The system embodiment described above is merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0238] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0239] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0240] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0241] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of the above units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0242] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0243] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0244] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0245] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A dynamic cross-modal hash retrieval method based on concept vector learning, characterized by: The method comprises the following steps: Obtain current label information of current multimodal data and historical label information of historical multimodal data; Based on the current label information and the Hadamard matrix, a concept vector matrix is ​​constructed; wherein the concept vectors in the concept vector matrix are used to represent that samples with the same label have the same and unchanged vectors; Based on the current label information and the historical label information, a data similarity matrix is ​​constructed; the data similarity matrix includes a first pairwise similarity matrix and a second pairwise similarity matrix, the first pairwise similarity matrix is ​​a pairwise similarity matrix between each sample in the current multimodal data, and the second pairwise similarity matrix is ​​a pairwise similarity matrix between each sample in the current multimodal data and the historical multimodal data, wherein the first pairwise similarity matrix is ​​composed of a current label matrix, and the second pairwise similarity matrix is ​​composed of the current label matrix and a historical label matrix, the current label matrix is ​​a label matrix corresponding to the current multimodal data, and the historical label matrix is ​​a label matrix corresponding to the historical multimodal data, the dimensions of the historical label matrix and the current label matrix are both determined by the number of categories at the current moment, and the historical label matrix and the current label matrix are both provided with a dimension expansion mechanism; Performing hash code learning based on the concept vector matrix and the data similarity matrix to obtain a target hash code, and storing the target hash code in a database; Performing hash function learning based on the target hash code to obtain a current hash function, and dynamically updating a historical hash function according to the current hash function to obtain a target hash function; Analyze the data to be queried according to the target hash function to obtain a hash code corresponding to the data to be queried; Calculate the similarity between the hash code to be queried and the target hash code in the database to obtain a target search result; The constructing of a concept vector matrix based on the current label information and the Hadamard matrix includes: Based on the current label information, assigning a column vector of the Hadamard matrix as a prototype code to a number of current category data in the current multimodal data; Based on the current sample label information corresponding to the current sample data in each of the current category data, generating a concept vector corresponding to each of the current sample label information in the concept space; The concept vector matrix is ​​constructed based on each of the concept vectors.

2. The method according to claim 1, characterized in that The generating, based on the current sample label information corresponding to the current sample data in each of the current category data, a concept vector corresponding to each of the current sample label information in the concept space includes: Based on the current sample label information corresponding to the current sample data in each current category data, the concept vector corresponding to each current sample label information is generated in the concept space by a concept vector calculation formula; the concept vector calculation formula is: in, Represented as a concept vector, Expressed as The number of prototype codes corresponding to the sample data, Expressed as The first sample data Prototype code, Represents the sequence number of the sample data, Indicates the serial number of the prototype code.

3. The method according to claim 1, characterized in that The constructing of a data similarity matrix based on the current tag information and the historical tag information includes: Calculating first similarity information between current sample data in the current multimodal data based on the current label information, and constructing a first pairwise similarity matrix according to the first similarity information; Based on the current label information and the historical label information, second similarity information between the current sample data in the current multimodal data and the historical sample data in the historical multimodal data is calculated, and the second pairwise similarity matrix is ​​constructed according to the second similarity information.

4. The method according to claim 1, wherein The performing hash code learning based on the concept vector matrix and the data similarity matrix to obtain a target hash code, and storing the target hash code in a database includes: Constructing a target optimization function based on the concept vector matrix and the data similarity matrix; Initialize the current hash code; Alternately optimizing the current hash code using the target optimization function to obtain the target hash code; The target hash code is stored in the database.

5. The method according to claim 1, wherein The performing hash function learning based on the target hash code to obtain a current hash function, and dynamically updating a historical hash function according to the current hash function to obtain a target hash function, includes: Based on the target hash code, the current multimodal data, and the historical multimodal data, a linear regression method is used to perform hash function learning to obtain the current hash function; The historical hash function is dynamically updated according to the current hash function to obtain the target hash function.

6. The method according to claim 1, characterized in that The similarity calculation between the hash code to be queried and the target hash code in the database to obtain a target search result includes: Calculating the Hamming distance between the hash code to be queried and the target hash code in the database to obtain a similarity calculation result; The target search result is obtained according to the similarity calculation result and the preset search rules.

7. Dynamic cross-modal hash retrieval system based on concept vector learning, characterized by: The system includes the following modules: A data acquisition module is used to obtain current label information of current multimodal data and historical label information of historical multimodal data; A concept vector matrix construction module is used to construct a concept vector matrix based on the current label information and the Hadamard matrix; wherein the concept vectors in the concept vector matrix are used to represent that samples with the same label have the same and unchanged vectors; A similarity matrix construction module is used to construct a data similarity matrix based on the current label information and the historical label information; the data similarity matrix includes a first pairwise similarity matrix and a second pairwise similarity matrix, the first pairwise similarity matrix is ​​a pairwise similarity matrix between each sample in the current multimodal data, and the second pairwise similarity matrix is ​​a pairwise similarity matrix between each sample in the current multimodal data and the historical multimodal data, wherein the first pairwise similarity matrix is ​​composed of a current label matrix, and the second pairwise similarity matrix is ​​composed of the current label matrix and a historical label matrix, the current label matrix is ​​a label matrix corresponding to the current multimodal data, and the historical label matrix is ​​a label matrix corresponding to the historical multimodal data, the dimensions of the historical label matrix and the current label matrix are both determined by the number of categories at the current moment, and the historical label matrix and the current label matrix are both provided with a dimension expansion mechanism; a hash code learning module, configured to perform hash code learning based on the concept vector matrix and the data similarity matrix to obtain a target hash code, and store the target hash code in a database; A hash function learning module is configured to obtain a current hash function based on the target hash code, and dynamically update a historical hash function according to the current hash function to obtain a target hash function; A hash code analysis module, configured to analyze the data to be queried according to the target hash function to obtain a hash code corresponding to the data to be queried; A hash retrieval module is used to calculate the similarity between the hash code to be queried and the target hash code in the database to obtain a target retrieval result; The concept vector matrix building module is specifically used to: Based on the current label information, assigning a column vector of the Hadamard matrix as a prototype code to a number of current category data in the current multimodal data; Based on the current sample label information corresponding to the current sample data in each of the current category data, generating a concept vector corresponding to each of the current sample label information in the concept space; The concept vector matrix is ​​constructed based on each of the concept vectors.

8. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images