A method for constructing a hash generation model of multimedia data and application thereof

Abstract

The application discloses a kind of multimedia data's hash generation model's construction method and application, multimedia data cross modal retrieval technical field, comprising: S1, receives multimedia data set;Wherein, multimedia data includes image, and text for describing image content, and carry with hierarchical label with hierarchical structure;S2, for each multimedia data, extract its modal feature, and obtain the membership relation matrix between any two adjacent layers in its hierarchical label, obtain the soft similarity label of each layer based on membership relation matrix and hierarchical label, and then obtain the hash code of each multimedia data;S3, with the modal feature of each multimedia data as input, the corresponding hash code is output, the machine learning model is trained, and the hash generation model is obtained.The application can make full use of the information in label hierarchical structure, generate hash code with discriminative power, greatly improve the accuracy of retrieval.

Description

Technical Field

[0001] This invention belongs to the field of multimedia data cross-modal retrieval technology, and more specifically, relates to a method for constructing and applying a hash generation model for multimedia data. Background Technology

[0002] With the rapid development of the internet, massive amounts of multimedia data, such as images and text, are uploaded, shared, and stored daily. Hashing is a method for solving multimedia data retrieval problems. This method primarily addresses scenarios where, given multimedia data, similar multimedia data is retrieved. It constructs a hash code for each multimedia data item in the database; the smaller the Hamming distance between the hash codes of two multimedia data items, the more similar they are. Based on the hash codes of multimedia data, retrieval systems can efficiently and quickly retrieve multimedia data while significantly reducing storage space.

[0003] Currently, hash generation methods are receiving increasing attention. However, existing hash generation methods neglect the hierarchical structure present in the dataset, resulting in a significant loss of semantic information in the generated hash codes. Researching cross-modal hash generation methods based on hierarchical labels is of great significance for achieving accurate and efficient retrieval on large-scale multimodal datasets, especially for achieving retrieval from coarse-grained to fine-grained. Summary of the Invention

[0004] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a method for constructing a hash generation model for multimedia data and its application. The purpose is to make full use of the information in the tag hierarchy to reduce the loss of semantic information in the process of mapping data into hash codes, generate hash codes with discriminative power, and improve the accuracy of retrieval.

[0005] To achieve the above objectives, the present invention provides a method for constructing a hash generation model for multimedia data, comprising the following steps:

[0006] S1. Receive a multimedia dataset; the multimedia data includes images and text describing the image content, and carries hierarchical labels with a hierarchical structure; the granularity of the category information carried by each layer of labels decreases from top to bottom, and is represented in vector form; the label L of the k-th layer of the multimedia data... k The i-th value in the table indicates whether the multimedia data belongs to the i-th category in the k-th layer; i = 1, 2, ..., c k ;c k The total number of categories in the k-th layer;

[0007] S2, for each multimedia data, extracting its modal feature, and obtaining the membership relation matrix between any two adjacent layers in its hierarchical label, obtaining the soft similar label of each layer based on the membership relation matrix and the hierarchical label, and then obtaining the hash code of each multimedia data;

[0008] wherein the modal feature comprises: image feature and text feature; the soft similar label S k+1 of the k+1th layer is calculated by k+1 +γ k+ 1 L k+1 ; u k+1 is the normalized result of v k+1 ; v k+1 = L k A k,k+1 + L k+1 ; A k,k+1 is the membership relation matrix between the kth layer and the k+1th layer in the hierarchical label, and the element in the ith row and the jth column represents whether the jth category of the k+1th layer is subordinate to the ith category of the kth layer; γ k+1 ∈(0,1] is the weight value under the k+1th layer;

[0009] S3, taking the modal feature of each multimedia data as input and the corresponding hash code as output, training the machine learning model to obtain a hash generation model.

[0010] Further preferably, the element in the ith row and the jth column of A k,k+1 is:

[0011]

[0012] wherein j k+1 represents the jth category of the k+1th layer; i k represents the ith category of the kth layer; k∈{1,2,…,K-1}, and K is the total number of layers of the hierarchical label.

[0013] Further preferably, the label L k of the kth layer of the multimedia data is a vector with a length of c k , and when the multimedia data belongs to the ith category in the kth layer, the ith value in the vector is 1, and the rest is 0.

[0014] Further preferably, the hash code of the multimedia data is obtained by solving the total objective function;

[0015] wherein the total objective function is:

[0016]

[0017]

[0018] wherein, η is a weight coefficient; r is the length of the hash code; N is the number of multimedia data in the multimedia data set; c K is the total number of classes in the Kth layer; O1 is a first objective function for maintaining the cross-layer membership relationship between classes in the hierarchical label; O2 is a second objective function for maintaining the fine-grained semantic similarity relationship between the multimedia data and the classes;

[0019] The first objective function is:

[0020]

[0021] The second objective function is:

[0022]

[0023] wherein, K is the total number of layers of the hierarchical label; β k is a weight value for measuring the importance of the membership relationship between the kth layer and the Kth layer classes; A k,K = A k,k+1 × A k+1,k+2 ×…× A K-1,K ; C k is the virtual center of the kth layer; C k is the virtual center of the Kth layer; α k is the confidence of the kth layer, used to weigh the distance of the hash code of the multimedia data to each layer virtual center; S k is the soft similar label of the kth layer of the multimedia data; B is the hash code of the multimedia data.

[0024] Further preferably, the construction method of the hash generation model further comprises: every time interval ΔT, re-collecting a multimedia data set, and performing the above steps S1-S2 on the new multimedia data set to obtain the modal features and hash codes of each multimedia data in the new multimedia data set, and inputting the modal features of each multimedia data of the new multimedia data set as input and the corresponding hash code as output, and incrementally training the hash generation model to update the hash generation model.

[0025] In a second aspect, the present application provides a hash generation method for multimedia data, comprising: inputting the modal features of the multimedia data into a hash generation model to obtain the corresponding hash code;

[0026] wherein, the hash generation model is constructed by the construction method of the hash generation model provided in the first aspect of the present application; the multimedia data includes images and / or texts for describing image content; the modal features include image features and / or text features.

[0027] In a third aspect, the present application provides a hash generation system for multimedia data, comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to execute the hash generation method provided in the second aspect of the present application.

[0028] In a fourth aspect, the present application provides a retrieval method for multimedia data, for retrieving multimedia data similar to a query multimedia data from a multimedia data set, comprising:

[0029] inputting the modal feature of the query multimedia data into the hash generation model to obtain the hash code of the query multimedia data, calculating the Hamming distance between the hash code of the query multimedia data and the hash code of each multimedia data in the multimedia data set, and sorting the obtained Hamming distances from small to large, and outputting the multimedia data in the multimedia data set corresponding to the first M Hamming distances; M>1;

[0030] wherein the query multimedia data comprises an image and / or text for describing the content of the image; the modal feature comprises an image feature and / or a text feature; the hash generation model is constructed by using the construction method of the hash generation model provided in the first aspect of the present application; and the hash code of each multimedia data in the multimedia data set is calculated by using steps S1-S2 in the construction method of the hash generation model provided in the first aspect of the present application.

[0031] In a fifth aspect, the present application provides a hash generation system for multimedia data, comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to execute the retrieval method for multimedia data provided in the fourth aspect of the present application.

[0032] In a sixth aspect, the present application further provides a computer readable storage medium, comprising a stored computer program, wherein when the computer program is run by a processor, the computer readable storage medium controls the device where the computer readable storage medium is located to execute one or more of the construction method of the hash generation model provided in the first aspect of the present application, the hash generation method provided in the second aspect of the present application, and the retrieval method provided in the fourth aspect of the present application.

[0033] Overall, through the above technical solutions conceived by the present application, the following beneficial effects can be achieved:

[0034] 1. The application provides a multimedia data hash generation model construction method, which is different from the traditional hard similarity label (only composed of 0 and 1) which can only represent whether the multimedia data is marked by a certain category, and the soft similarity label is introduced, the soft similarity label is based on a tree-shaped label hierarchy, which not only emphasizes the category marked by the hard similarity label, but also represents the fine-grained similarity between the sample and the sibling categories of the categories; the fine-grained similarity between the multimedia data and the categories of the current layer is obtained by considering the category membership relationship between the adjacent two layers, and the fine-grained similarity between the multimedia data and the categories of the current layer is used as the soft similarity label of the multimedia data in the current layer, so that the fine-grained semantic information between the multimedia data and the categories can be more accurately represented. Based on this, the information in the label hierarchy is fully utilized, a high-quality hash code is generated for each multimedia data in the multimedia data set, and the hash generation model is trained by taking the modal features of the multimedia data as input; the constructed hash generation model can reduce the loss of semantic information in the process of mapping the multimedia data into a hash code, and greatly improves the accuracy of retrieval.

[0035] 2. Further, the hash generation model construction method provided by the application can maintain the cross-layer membership relationship between the categories in the hierarchical label and the fine-grained semantic similarity relationship between the multimedia data and the categories based on the soft similarity label to obtain the hash code of the multimedia data, so that the label hierarchy information can be retained, the information loss in the hash code learning process can be reduced, and the precision and recall rates in coarse-grained retrieval and fine-grained retrieval can be improved.

[0036] 3. Further, the hash generation model construction method provided by the application performs incremental training on the hash generation model every certain period of time to further improve the generalization ability and accuracy of the model.

[0037] 4. The application learns unified hash codes from different modalities, which not only allows the hash codes to integrate information of multiple modalities, but also can easily extend the method to more modalities, and has a wide range of applications. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 The hash generation model construction method for multimedia data provided for the embodiment 1 of the application is shown in the flowchart;

[0039] Figure 2 The influence diagram of the label hierarchy structure on the mapping of the multimedia data provided for the embodiment 1 of the application is shown in the flowchart;

[0040] Figure 3 The hash code learning flowchart based on the label hierarchy structure provided for the embodiment 1 of the application is shown in the flowchart;

[0041] Figure 4A flow chart of an online retrieval method of multimedia data provided for Embodiment 4 of the present application. DETAILED DESCRIPTION

[0042] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. In addition, the technical features involved in the various embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0043] Embodiment 1,

[0044] In order to achieve the above-mentioned purpose, the present application provides a construction method of a hash generation model of multimedia data, as shown in Figure 1 , comprising the following steps:

[0045] S1, receiving a multimedia data set; wherein the multimedia data includes images and texts (i.e. image-text pairs) for describing image content, and carries hierarchical labels with a hierarchical structure, which is used for semantic category division of the multimedia data; the category information granularity carried by each layer label is successively refined from top to bottom, and is represented in the form of a vector; the i-th value in the label L k of the k-th layer of the multimedia data represents whether the multimedia data belongs to the i-th category in the k-th layer; i = 1, 2, …, c k ; c k is the total number of categories in the k-th layer;

[0046] It should be noted that the multimedia data set can be a data set collected by a user, or a general data set, such as the FashionVC data set, the Ssense data set, etc.

[0047] S2, for each multimedia data, extracting its image features and text features, and obtaining the membership relationship matrix between any two adjacent layers in its hierarchical label, obtaining the soft similar label of each layer based on the membership relationship matrix and the hierarchical label, and further obtaining the hash code of each multimedia data;

[0048] Wherein, the soft similar label S k+1 of the k+1-th layer is u k+1 +γ k+1 L k+1 ; u k+1 is the normalized result of v k+1 , for example, the 2-norm normalization of v k+1 , the i-th element k+1 of u is vk+1 The i-th element. k+1 =L k A k,k+1 +L k+1 A k,k+1 This is a membership matrix between the k-th and (k+1)-th levels in the hierarchical labeling system. The element in the i-th row and j-th column indicates whether the j-th category of the (k+1)-th level belongs to the i-th category of the k-th level; γ k+1 ∈(0,1] represents the weight value at the (k+1)th layer, used to highlight the category of multimedia data at the (k+1)th layer; preferably, γ k+1 The value is set to 1 to balance the fine-grained semantic information between multimedia data and its (k+1)th level category and its siblings.

[0049] Taking a two-layer hierarchical structure as an example, the parent-child relationship between the first-level category (layer k) and the second-level category (layer k+1) is clarified through the membership matrix. It is determined whether the second-level categories have a common parent class, thereby judging which second-level categories are siblings or unrelated classes, and then obtaining the fine-grained similarity between multimedia data and the second-level categories, that is, the soft similarity label of the k+1 layer.

[0050] Specifically, A k,k+1 The element in the i-th row and j-th column is:

[0051]

[0052] Where, j k+1 Represents the j-th category in the (k+1)-th layer; i k Let represent the i-th category of the k-th layer; k∈{1,2,…,K-1}, where K is the total number of layers in the hierarchy labels. Clearly, the categories in the k-th layer are the most granular.

[0053] In one alternative implementation, such as Figure 2 As shown, the first level of categories includes "accessories" and "bags," while the second level includes "gloves," "socks," "backpacks," and "handbags," forming a 2×4 membership matrix.

[0054] Furthermore, the label L of the k-th layer of multimedia data k For a length of c k The vector is such that when multimedia data belongs to the i-th category in the k-th layer, the i-th value in the vector is 1, and the rest are 0.

[0055] Further, the image features of the multimedia data can be extracted by using image feature extraction models such as VGG, GIST, SIFT, Resnet, and ViT; and the text features of the multimedia data can be extracted by using text feature models such as BoW, TF-IDF, Word2Vec, GloVe, and BERT. Specifically, in an optional implementation, the FashionVC dataset is used as the multimedia dataset, the VGG16 network is used to extract 4096-dimensional image features, and the BoW model is used to extract 2685-dimensional text features. In another optional implementation, the Ssense dataset is used as the multimedia dataset, the VGG16 network is used to extract 4096-dimensional image features, and the BoW model is used to extract 4945-dimensional text features.

[0056] Further, by solving the total objective function, the hash code of the multimedia data is obtained; specifically, the virtual centers of the hierarchical categories and the hash code of the multimedia data are iteratively learned according to the soft similarity labels and the hierarchical labels. The specific process is as shown in Figure 3

[0057] The first objective function is used to embed the similarity of the multimedia data and the category labels of each layer in the label hierarchy into the hierarchical virtual centers, for maintaining the cross-layer membership relationship between the categories in the hierarchical labels.

[0058] The second loss function is used to embed the fine-grained semantic similarity relationship into the hash code of the multimedia data layer by layer, for maintaining the fine-grained semantic similarity relationship between the multimedia data and the categories.

[0059] The first objective function and the second objective function are integrated to obtain the total objective function, which can reduce the loss of semantic information in the process of mapping the multimedia data into the hash code, so as to generate high-quality hash codes.

[0060] The first objective function is:

[0061]

[0062] The second objective function is:

[0063]

[0064] wherein K is the total number of hierarchical labels; β k is a weight value for measuring the importance of the membership relationship between the k th layer and the K th layer; r is the length of the hash code; A k is the membership value of the k th layer; and A K is the membership value of the K th layer. k k,K k,k+1 k+1,k+2 K-1,K ​​​​​;C k is the virtual center of the kth layer; C K is the virtual center of the kth layer; a k is the confidence of the kth layer, used to weigh the distance of the hash code of the multimedia data to the virtual center of each layer; S k is the soft similar label of the kth layer of the multimedia data; B is the hash code of the multimedia data.

[0065] The total objective function is:

[0066]

[0067]

[0068] wherein, η is a weight coefficient; N is the number of multimedia data in the multimedia data set; c K is the total number of categories in the kth layer.

[0069] It should be noted that ||·|| in the first objective function represents the F norm used for calculating the matrix; ||·|| in the second objective function represents the 2 norm used for calculating the vector.

[0070] In an optional embodiment, the discrete optimization algorithm, the relaxation strategy and the semi-relaxation strategy are used to solve the total objective function to obtain the hash code of the multimedia data. In order to reduce the quantization error and efficiently learn the hash code, preferably, the discrete optimization algorithm (DCC algorithm) is used to iteratively and alternately learn the hash code of the multimedia data and the hierarchical virtual center.

[0071] S3, taking the image features and the text features of each multimedia data as input and the corresponding hash code as output, training the machine learning model to obtain a hash generation model.

[0072] Wherein, the machine learning model can be a linear regression model, a neural network model (such as a convolutional neural network CNN, a recurrent neural network RNN), etc.

[0073] Specifically, according to the image feature matrix, the text feature matrix and the hash code matrix of each multimedia data, the hash functions of each modality (the hash function of the image modality and the hash function of the text modality) are learned to obtain the hash generation model;

[0074] The hash generation model is specifically:

[0075] W m =F m (G m +I) -1

[0076] Wherein,

[0077]

[0078]

[0079] W m is the projection matrix of the mth modality of the multimedia data; I is the unit matrix; B is the hash code; X m is the modality feature under the mth modality of the multimedia data; the modality includes an image modality and a text modality.

[0080] Further, in an optional embodiment, the method for constructing the hash generation model further comprises: every ΔT, reacquiring a multimedia data set, and performing the steps S1-S2 on the new multimedia data set to obtain the image feature, the text feature and the hash code of each multimedia data in the new multimedia data set, and taking the image feature and the text feature of each multimedia data in the new multimedia data set as input and the corresponding hash code as output, training the hash generation model to update the hash generation model. In this embodiment, ΔT is one hour.

[0081] In summary, the hash generation model constructed by the application can generate high-quality hash codes and improve the retrieval performance, especially in the query scene from coarse granularity to fine granularity.

[0082] Embodiment 2,

[0083] A hash generation method of multimedia data comprises: inputting a modality feature of the multimedia data into a hash generation model to obtain a corresponding hash code.

[0084] The hash generation model is constructed by the method for constructing the hash generation model provided in Embodiment 1 of the application; the multimedia data comprises an image and / or text for describing the image content; when the multimedia data only comprises the image, the modality feature is an image feature; when the multimedia data only comprises the text, the modality feature is a text feature; when the multimedia data comprises both the image and the text, the modality feature comprises the image feature and the text feature. The modality feature can also be extracted by the feature extraction method provided in Embodiment 1 of the application.

[0085] Specifically, the multimedia data to be queried is acquired, and the modality feature of the multimedia data to be queried is mapped by a hash function of the same modality in the hash generation model to obtain the hash code of the multimedia data to be queried.

[0086] The related technical solution is the same as that in Embodiment 1, and will not be repeated here.

[0087] Embodiment 3,

[0088] A hash generation system of multimedia data, comprising a memory and a processor, the memory stores a computer program, and the processor executes the computer program to execute the hash generation method provided in Embodiment 2 of the present application.

[0089] The related technical solutions are the same as those in Embodiment 2, and will not be repeated here.

[0090] Embodiment 4,

[0091] A retrieval method of multimedia data is used to retrieve multimedia data similar to the multimedia data to be queried in a multimedia data set, comprising:

[0092] The modal feature of the multimedia data to be queried is input into the hash generation model to obtain the hash code of the multimedia data to be queried, the Hamming distance between the hash code of the multimedia data to be queried and the hash code of each multimedia data in the multimedia data set is calculated, and the obtained Hamming distances are sorted from small to large, and the multimedia data in the multimedia data set corresponding to the first M Hamming distances is output; M≥1, when M is 1, the multimedia data most similar to the multimedia data to be queried in the multimedia data set is selected; in this embodiment, M is 1000 or 2000.

[0093] Wherein, the multimedia data to be queried includes images and / or text for describing image content; the modal feature includes image features and / or text features; when the multimedia data only includes images, the above-mentioned modal feature is image feature; when the multimedia data only includes text, the above-mentioned modal feature is text feature; when the multimedia data includes both images and text, the above-mentioned modal feature includes image features and text features. The modal feature can also be extracted by the feature extraction method provided in Embodiment 1 of the present application. The hash generation model is constructed by the construction method of the hash generation model provided in Embodiment 1 of the present application; the hash code of each multimedia data in the multimedia data set is calculated by steps S1-S2 in the construction method of the hash generation model provided in Embodiment 1 of the present application.

[0094] It should be noted that the present application is suitable for both offline and online scenarios. Specifically, taking the online scenario as an example, the flowchart of the online retrieval method of multimedia data is as shown in Figure 4 , which specifically includes:

[0095] A1, obtain training data, and divide the training data into T rounds to simulate the data streaming in the online scenario. In this embodiment, T is [N / 2000]; N is the total number of multimedia data in the multimedia data set used for training, specifically, when the multimedia data set is FashionVC, T is 8; when the multimedia data set is Ssense, T is 6.

[0096] A2. In each round, obtain the image feature matrix, the text feature matrix, the hierarchical label matrix and the cross-layer membership matrix of the data stream of the round;

[0097] A3. Construct a hierarchical soft similar label matrix according to the hierarchical label matrix and the cross-layer membership matrix;

[0098] A4. Iteratively learn the virtual center of the hierarchical category and the hash code of the data stream according to the soft similar label matrix;

[0099] A5. Store the image feature matrix, the text feature matrix and the hash code matrix of the data stream of the round into the database;

[0100] A6. Learn the hash function of each modality according to the image feature matrix, the text feature matrix and the hash code matrix of the database;

[0101] A7. Obtain a to-be-queried sample, and obtain the hash code of the to-be-queried sample after the feature vector of the to-be-queried sample is mapped through the hash function of the same modality.

[0102] A8. Calculate the Hamming distance between the hash code of the to-be-queried sample and the hash code matrix of the database, sort the retrieved similar samples in ascending order of the Hamming distance, and return the result.

[0103] A9. Repeat A2-A8 until the Tth round of training is completed.

[0104] The hash code of the data stream is learned and stored in the database through the soft similar label matrix and the learned hierarchical virtual center, and when searching, the Hamming distance between the hash code generated by the hash function and the hash code in the database can be directly compared, without the need to recalculate the hash code of the data stream, thereby greatly reducing the computational overhead.

[0105] The related technical solutions are the same as those of embodiment 1, and will not be repeated here.

[0106] Embodiment 5,

[0107] A multimedia data hash generation system, comprising a memory and a processor, the memory stores a computer program, and the processor executes the computer program to execute the retrieval method of the multimedia data provided in embodiment 4 of the application.

[0108] The related technical solutions are the same as those of embodiment 4, and will not be repeated here.

[0109] Embodiment 6,

[0110] A computer readable storage medium, comprising a stored computer program, wherein the computer program, when executed by a processor, controls a device where the storage medium is located to perform one or more of the construction method of the hash generation model provided in Embodiment 1, the hash generation method provided in Embodiment 2, and the retrieval method provided in Embodiment 4.

[0111] The related technical solutions are the same as those in Embodiment 1, Embodiment 2, and Embodiment 4, and are not described herein.

[0112] Those skilled in the art can easily understand that the above description is only the preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, and improvement within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for constructing a hash generation model for multimedia data, characterized in that, Includes the following steps: S1. Receive a multimedia dataset; the multimedia data includes images and text describing the image content, and carries hierarchical labels with a hierarchical structure; the granularity of the category information carried by each layer of labels decreases from top to bottom, and is represented in vector form; the label L of the k-th layer of the multimedia data... k The i-th value in the table indicates whether the multimedia data belongs to the i-th category in the k-th layer; i = 1, 2, ..., c k c k The total number of categories in the k-th layer; S2. For each multimedia data, extract its modal features and obtain the membership matrix between any two adjacent layers in its hierarchical labels. Based on the membership matrix and hierarchical labels, obtain the soft similarity label of each layer, and then obtain the hash code of each multimedia data. The modal features include: image features and text features; and the soft similarity label S of the (k+1)th layer. k+1 =u k+1 +γ k+ 1 L k+1 ;u k+1 For v k+1 The normalization result; v k+1 =L k A k,k+1 +L k+1 A k,k+1 This is a membership matrix between the k-th and (k+1)-th levels in the hierarchical labeling system. The element in the i-th row and j-th column indicates whether the j-th category of the (k+1)-th level belongs to the i-th category of the k-th level; γ k+1 ∈(0,1] represents the weight value at the (k+1)th layer; S3. Using the modal features of various multimedia data as input and the corresponding hash codes as output, train the machine learning model to obtain the hash generation model.

2. The method for constructing a hash generation model according to claim 1, characterized in that, A k,k+1 The element in the i-th row and j-th column is: Where, j k+1 Represents the j-th category in the (k+1)-th layer; i k This represents the i-th category in the k-th layer; k∈{1,2,…,K-1}, where K is the total number of layers in the hierarchy labels.

3. The method for constructing a hash generation model according to claim 1, characterized in that, Multimedia data layer k label L k For a length of c k The vector is such that when multimedia data belongs to the i-th category in the k-th layer, the i-th value in the vector is 1, and the rest are 0.

4. The method for constructing a hash generation model according to claim 1, characterized in that, The hash code of the multimedia data is obtained by solving the overall objective function; The overall objective function is: s.t.B∈{-1,1} r×1 , B is the hash code of the multimedia data; r is the length of the hash code; N is the number of multimedia data in the multimedia dataset; C K c is the virtual center of the Kth layer; K C is the total number of categories in the Kth layer; k η is the virtual center of the k-th layer; η is the weight coefficient; O1 is the first objective function, used to maintain the cross-layer membership relationship between categories in the hierarchical labels; O2 is the second objective function, used to maintain the fine-grained semantic similarity relationship between multimedia data and categories; The first objective function is: The second objective function is: K is the total number of hierarchical labels; β k A is a weight value used to measure the importance of the membership relationship between categories in layer k and layer K; k,K =A k,k+1 ×A k+1,k+2 ×...×A K-1,K ;α k S represents the confidence level of the k-th layer, used to weigh the distance from the hash code of the multimedia data to the virtual center of each layer; k For the k-th layer of multimedia data, use soft similarity tags.

5. The method for constructing a hash generation model according to any one of claims 1-4, characterized in that, Also includes: Every time interval ΔT, a new multimedia dataset is collected, and steps S1-S2 are performed on the new multimedia dataset to obtain the modal features and hash codes of each multimedia data in the new multimedia dataset. The hash generation model is incrementally trained using the modal features of each multimedia data in the new multimedia dataset as input and the corresponding hash codes as output, so as to update the hash generation model.

6. A method for generating hashes for multimedia data, characterized in that, include: The modal features of multimedia data are input into the hash generation model to obtain the corresponding hash code; The hash generation model is constructed using the hash generation model construction method described in any one of claims 1-5; the multimedia data includes images and / or text used to describe the content of the images; the modal features include image features and / or text features.

7. A hash generation system for multimedia data, characterized in that, include: A memory and a processor, wherein the memory stores a computer program, and the processor executes the hash generation method of claim 6 when executing the computer program.

8. A method for retrieving multimedia data, characterized in that, Used to retrieve multimedia data similar to the multimedia data to be queried in a multimedia dataset, including: The modal features of the multimedia data to be queried are input into the hash generation model to obtain the hash code of the multimedia data to be queried. The Hamming distance between the hash code of the multimedia data to be queried and the hash code of each multimedia data in the multimedia dataset is calculated. The obtained Hamming distances are sorted from smallest to largest, and the multimedia data in the multimedia dataset corresponding to the first M Hamming distances are output; M≥1. Wherein, the multimedia data to be queried includes images and / or text used to describe the content of the images; the modal features include image features and / or text features; the hash generation model is constructed using the construction method of the hash generation model according to any one of claims 1-5; the hash code of each multimedia data in the multimedia dataset is calculated using steps S1-S2 in the construction method of the hash generation model according to any one of claims 1-5.

9. A hash generation system for multimedia data, characterized in that, include: A memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the retrieval method of claim 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is run by a processor, it controls the device where the storage medium is located to perform one or more of the following: the method for constructing a hash generation model according to any one of claims 1-5, the hash generation method according to claim 6, and the retrieval method according to claim 8.

Images