Image-text matching method in multi-object scene based on heterogeneous information interaction

By constructing a joint hash code matrix and a hash function learning formula, and using an alternating update algorithm, the problem of image-text matching in multi-object scenarios was solved, achieving a higher matching accuracy.

CN119003802AActive Publication Date: 2024-11-22HARBIN UNIV OF SCI & TECH
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411082605.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-08
Publication Date
2024-11-22
Estimated Expiration
2044-08-08

AI Technical Summary

Technical Problem

Existing image-text matching methods perform poorly in multi-object scenarios, making it difficult to achieve accurate image-text matching.

Method used

A heterogeneous information interaction-based approach is adopted. By constructing a joint hash code matrix learning formula and a hash function learning formula, and solving them using an alternating update algorithm, the independent latent representation spaces of images and text are learned respectively. Through heterogeneous information interaction constraints, a more discriminative joint hash code matrix and hash function are generated.

Benefits of technology

It improves the accuracy of image-text matching in multi-object scenarios by preserving the unique information of images and text and leveraging heterogeneous information interaction to enhance the proximity and matching accuracy of latent representations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure BDA0004984198810000021
    Figure BDA0004984198810000021
  • Figure BDA0004984198810000041
    Figure BDA0004984198810000041
  • Figure BDA0004984198810000042
    Figure BDA0004984198810000042
Patent Text Reader

Abstract

The invention discloses an image-text matching method in a multi-object scene based on heterogeneous information interaction, and belongs to the field of image-text matching in multimedia information processing. The method comprises the following steps: dividing a data set into a training set and a test set; then, constructing a joint hash code matrix learning formula aiming at a multi-object data image-text matching problem; then, solving the joint Hash code matrix learning formula by utilizing an alternating updating algorithm 1 to obtain a joint Hash code matrix B; secondly, constructing a hash function learning formula aiming at a multi-object data image-text matching problem; then, the Hash function learning formula is solved through an alternating updating algorithm 2, and Hash functions F1 and F2 are obtained; and finally, testing the test set by using the combined hash code matrix B and the hash functions F1 and F2 obtained in the above steps, and calculating the average accuracy of image-text matching in a multi-object scene. Compared with other methods, the average accuracy of the method is higher.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of image-text matching in multimedia information processing, and in particular relates to an image-text matching method in a multi-object scenario based on heterogeneous information interaction. Background Art

[0002] In the field of multimedia information processing, cross-modal matching refers to the matching between different modalities. Usually, one modality is used as a query sample, and the data of another modality is matched through an algorithm. Images and texts are widely used and easy to obtain data modal types, so image-text matching is one of the main research directions of cross-modal matching. Since most social media have limited content, people will share image and text information by information splicing and fusion. For example, when posting on WeChat Moments, people tend not to post multiple pictures and distribute corresponding text descriptions separately, but prefer to splice photos and add one or more rich text descriptions, so as to share as much information as possible in limited content. This results in a paragraph of text and a picture not only containing information about a certain item, but also containing complex information about multiple items. This type of image and text presentation containing multiple items has become the first choice for most people when sharing. Due to the rich information, the processing and matching problems of this type of data will be more complicated. Therefore, how to accurately match images and texts in multi-object scenarios is a problem with application value. However, existing image-text matching methods perform poorly in this scenario, and effective novel technical methods are needed to solve the image-text matching problem in multi-object scenarios. Summary of the invention

[0003] In order to solve the above problems, the present invention provides a method for image-text matching in a multi-object scenario based on heterogeneous information interaction, the method comprising the steps of:

[0004] Divide the dataset into training set and test set.

[0005] Construct a joint hash code matrix learning formula for the multi-object data graph-text matching problem.

[0006] The joint hash code matrix learning formula is solved using the alternating update algorithm 1 to obtain the joint hash code matrix B.

[0007] Construct a hash function learning formula for the multi-object data graph-text matching problem.

[0008] The hash function learning formula is solved using the alternating update algorithm 2 to obtain hash functions F1 and F2.

[0009] The joint hash code matrix B and hash functions F1 and F2 obtained in the above steps are used to test the test set and calculate the average accuracy of image-text matching in multi-object scenarios.

[0010] Furthermore, the joint hash code matrix learning formula is:

[0011]

[0012] stE1,E2≥0,B∈{-1,1} r×n ,

[0013] in, is the multi-object image data matrix, is a multi-object text data matrix, where d1 and d2 are the dimensions of multi-object image and text data, and n is the number of samples. is the multi-object image and text data matrix to be processed by kernelization, where q is the dimension of the kernelized data. For a column vector x at a certain position in the multi-object image and text data matrix, the column vector at the corresponding position in the kernelized multi-object image and text matrix is in is the square of the 2-norm of the vector, is the q column vectors randomly extracted from the corresponding data matrix without duplication, q is also the dimension of the kernelized data, is the kernel width, x i is the i-th column vector of the corresponding data matrix, g j for The jth column vector in R, ||·||2 is the 2-norm of the vector, U1, U2∈R q×r are the latent factor matrices of multi-object images and text data, respectively, where r is the hash code length, L∈R c×n is the category labeling matrix. When the jth data contains the i-th object, L ij =1, otherwise L ij = 0, where c is the number of object types, is the normalized category labeling matrix, that is l i is the i-th column vector of the category label matrix, Γ∈R c×n is the index matrix, when L ij =1,Γ ij =1, otherwise Γ ij = -1, ⊙ is the symbol of Hadamard product, E1, E2∈R c×n are the interval element matrices of multi-object images and text data, respectively, which are used to strengthen the constraint of heterogeneous data being far away, P1, P2∈R c×r are the information projection matrices of multi-object images and text data, V1, V2∈R r×n is an independent latent representation space for multi-object images and texts, For heterogeneous information interaction, Used to guide heterogeneous information interaction, Q1, Q2∈R r×r are projection matrices with orthogonal constraints for multi-object images and text data, B∈{-1,1} r×n is a joint hash code matrix whose elements are either -1 or 1, is the square of the matrix F-norm, I r is the identity matrix of dimension r, and λ1, λ2, α, β1, β2, and μ are trade-off parameters.

[0014] Further, the alternating update algorithm 1 is summarized as follows:

[0015]

[0016]

[0017] Furthermore, the hash function learning formula is:

[0018]

[0019] stB∈{-1,1} r×n

[0020] in, is the normalized category labeling matrix, F1, F2∈R r×q are the hash functions for multi-object images and text data, respectively, where r is the hash code length, q is the dimension of the kernelized data mentioned in the joint hash code matrix learning formula, and μ1 and μ2 are trade-off parameters.

[0021] Furthermore, the alternating update algorithm 2 is summarized as follows:

[0022]

[0023] The present invention provides a method for image-text matching in a multi-object scenario based on heterogeneous information interaction, which has the following advantages:

[0024] (1) The method learns the independent latent representation spaces of images and texts respectively in the joint hash code matrix learning formula, retaining the unique information of images and texts, thereby making the latent representation space more discriminative.

[0025] (2) The method uses heterogeneous information interaction in the joint hash code matrix learning formula to link the independent latent representation spaces of images and texts, making the latent representations of the two modalities closer. At the same time, it can integrate multi-object information into the latent representation of each modality. With the help of the latent representations of the above modalities, a more discriminative joint hash code matrix B can be learned.

[0026] (3) The method imposes heterogeneous information interaction constraints on the hash function in the hash function learning formula, so that the encoding features obtained by hash functions F1 and F2 when projecting images and texts into the Hamming space are closer to the joint hash code matrix B, thereby improving the matching accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0028] Figure 1 It is a flow chart of the image-text matching method in a multi-object scenario based on heterogeneous information interaction provided by the present invention; DETAILED DESCRIPTION

[0029] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, the description of well-known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present invention.

[0030] Exemplary Methods

[0031] like Figure 1 The present invention provides a method for image-text matching in a multi-object scenario based on heterogeneous information interaction, and the method steps are as follows:

[0032] Step S110: Divide the data set into a training set and a test set.

[0033] Step S120: construct a joint hash code matrix learning formula for the multi-object data image-text matching problem.

[0034] The joint hash code matrix learning formula is as follows:

[0035]

[0036] stE1,E2≥0,B∈{-1,1} r×n ,

[0037] in, is the multi-object image data matrix, is a multi-object text data matrix, where d1 and d2 are the dimensions of multi-object image and text data, and n is the number of samples. is the multi-object image and text data matrix to be processed by kernelization, where q is the dimension of the kernelized data. For a column vector x at a certain position in the multi-object image and text data matrix, the column vector at the corresponding position in the kernelized multi-object image and text matrix is in is the square of the 2-norm of the vector, is the q column vectors randomly extracted from the corresponding data matrix without duplication, q is also the dimension of the kernelized data, is the kernel width, x i is the i-th column vector of the corresponding data matrix, g j for The jth column vector in R, ||·||2 is the 2-norm of the vector, U1, U2∈R q×r are the latent factor matrices of multi-object images and text data, respectively, where r is the hash code length, L∈R c×n is the category labeling matrix. When the jth data contains the i-th object, L ij =1, otherwise L ij = 0, where c is the number of object types, is the normalized category labeling matrix, that is l i is the i-th column vector of the category label matrix, Γ∈R c×n is the index matrix, when L ij =1,Γ ij =1, otherwise Γ ij = -1, ⊙ is the symbol of Hadamard product, E1, E2∈R c×n are the interval element matrices of multi-object images and text data, respectively, which are used to strengthen the constraint of heterogeneous data being far away, P1, P2∈R c×r are the information projection matrices of multi-object images and text data, V1, V2∈R r×n is an independent latent representation space for multi-object images and texts, For heterogeneous information interaction, Used to guide heterogeneous information interaction, Q1, Q2∈R r×r are projection matrices with orthogonal constraints for multi-object images and text data, B∈{-1,1} r×n is a joint hash code matrix whose elements are either -1 or 1, is the square of the matrix F-norm, I r is the identity matrix of dimension r, and λ1, λ2, α, β1, β2, and μ are trade-off parameters.

[0038] Step S130: Solve the joint hash code matrix learning formula using the alternating update algorithm 1 to obtain the joint hash code matrix B.

[0039] The alternating update algorithm 1 is summarized as follows:

[0040]

[0041]

[0042] Step S140: construct a hash function learning formula for the multi-object data graph-text matching problem.

[0043] The hash function learning formula is as follows:

[0044]

[0045] stB∈{-1,1} r×n

[0046] in, is the normalized category labeling matrix, F1, F2∈R r×q are the hash functions for multi-object images and text data, respectively, where r is the hash code length, q is the dimension of the kernelized data mentioned in the joint hash code matrix learning formula, and μ1 and μ2 are trade-off parameters.

[0047] Step S150: Solve the hash function learning formula using the alternating update algorithm 2 to obtain hash functions F1 and F2.

[0048] The alternating update algorithm 2 is summarized as follows:

[0049]

[0050] Step S160: Use the joint hash code matrix B and hash functions F1 and F2 obtained in the above steps to test the test set and calculate the average accuracy of image-text matching in the multi-object scene.

[0051] Through this implementation, the data set is first divided into a training set and a test set, and then a joint hash code matrix learning formula for the multi-object data image-text matching problem is constructed, and the alternating update algorithm 1 is used to solve the joint hash code matrix learning formula to obtain the joint hash code matrix B, and then a hash function learning formula for the multi-object data image-text matching problem is constructed, and the alternating update algorithm 2 is used to solve the hash function learning formula to obtain hash functions F1 and F2. Finally, the joint hash code matrix B and hash functions F1 and F2 obtained in the above steps are used to test the test set and calculate the average accuracy of image-text matching in the multi-object scenario.

[0052] To further illustrate, assuming that a multi-object image-text matching dataset is subjected to image-text matching according to this embodiment, a result with an average accuracy higher than that of most methods will be obtained.

[0053] Detailed description of the invention Results

[0054] This implementation uses a publicly available multi-object image-text matching dataset, the details of which are described as follows:

[0055] The COCO dataset consists of 122,218 text-image pairs with a total of 80 categories. These text-image pairs belong to at least one of the categories. The images are represented by 4096-dimensional features extracted by the Caffe framework, and the texts are represented by 2000-dimensional features extracted using the bag-of-words model.

[0056] In order to verify the superiority of this implementation (Ours), this implementation is compared with several existing image-text matching methods, including SCRATCH, LEMON, BATCH, HCCH and other methods. The average accuracy (mAP) of the above methods using hash codes of different lengths for image-text matching on the COCO dataset will be compared. The specific data comparison is shown in Table 1.

[0057] Table 1 Average accuracy mAP (%) of image-text matching in COCO dataset

[0058] Hash code length SCRATCH LEMON BATCH HCCH Ours 16 50.32 52.08 42.66 57.05 58.24 32 54.01 52.56 43.70 57.22 59.49 64 51.76 53.29 43.92 57.27 60.56 128 47.69 53.30 44.50 57.89 60.58

[0059] By comparing the data in the above two tables, we can see that Ours achieves the best performance in the selected hash code length, significantly improving the average accuracy of image-text matching in multi-object scenarios. The quantitative results fully demonstrate the superiority of Ours, because Ours retains the unique information of image and text modalities; Ours uses heterogeneous information interaction to communicate the latent representation space of images and texts, promotes the alignment of the two, and integrates multi-object information into the latent representation space to better solve the image-text matching problem in multi-object scenarios; the latent representation space learned using the above two strategies is more discriminative, which helps to generate a high-quality joint hash code matrix; Ours imposes heterogeneous information interaction constraints on the hash function learning formula, so that the encoding features obtained by the hash function projecting feature data into the Hamming space are closer to the joint hash code matrix. A large number of experiments show that this method is superior to most existing methods. The parameters of this embodiment are set as follows: q=1500, λ1=0.7, λ2=0.3, α=20, β1=0.45, β2=0.55, μ=5e-6, μ1=3e-6, μ2=5e-6, η=5e-6.

[0060] This embodiment proposes a method for image-text matching in multi-object scenes based on heterogeneous information interaction, which is used for image-text matching in multi-object scenes. The method learns the potential representation spaces of images and texts separately to retain the exclusive information of the modality, and uses heterogeneous information interaction to embed multi-object information into the potential representation space to solve the image-text matching problem in multi-object scenes. The alternating update algorithm 1 is used to solve the constructed joint hash code matrix learning formula to obtain the joint hash code matrix B, and then the alternating update algorithm 2 is used to solve the constructed hash function learning formula to obtain hash functions F1 and F2. Finally, the joint hash code matrix B and hash functions F1 and F2 are used to test the test set and calculate the average accuracy of image-text matching in multi-object scenes. The experimental results on the public data set COCO show that this embodiment has a higher average matching accuracy than other methods and has better superiority.

[0061] It should be understood that the above specific embodiments of the present invention are only used to illustrate or explain the principles of the present invention, and do not constitute a limitation of the present invention. Therefore, any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and scope of the present invention should be included in the protection scope of the present invention. In addition, the appended claims of the present invention are intended to cover all changes and modifications that fall within the scope and boundaries of the appended claims, or the equivalent forms of such scope and boundaries.

Claims

1. A method for image-text matching in a multi-object scene based on heterogeneous information interaction, the method comprising the steps of: Divide the data set into training set and test set; Construct a joint hash code matrix learning formula for multi-object data image-text matching problem; The joint hash code matrix learning formula is: in, is the multi-object image data matrix, is a multi-object text data matrix, where d1 and d2 are the dimensions of multi-object image and text data, and n is the number of samples. is the multi-object image and text data matrix to be processed by kernelization, where q is the dimension of the kernelized data. For a column vector x at a certain position in the multi-object image and text data matrix, the column vector at the corresponding position in the kernelized multi-object image and text matrix is in is the square of the 2-norm of the vector, is the q column vectors randomly extracted from the corresponding data matrix without duplication, q is also the dimension of the kernelized data, is the kernel width, x i is the i-th column vector of the corresponding data matrix, g j for The jth column vector in R, ||·||2 is the 2-norm of the vector, U1, U2∈R q×r are the latent factor matrices of multi-object images and text data, respectively, where r is the hash code length, L∈R c×n is the category labeling matrix. When the jth data contains the i-th object, L ij =1, otherwise L ij = 0, where c is the number of object types, is the normalized category labeling matrix, that is l i is the i-th column vector of the category label matrix, Γ∈R c×n is the index matrix, when L ij =1,Γ ij =1, otherwise Γ ij = -1, ⊙ is the symbol of Hadamard product, E1, E2∈R c×n are the interval element matrices of multi-object images and text data, respectively, which are used to strengthen the constraint of heterogeneous data being far away, P1, P2∈R c×r are the information projection matrices of multi-object images and text data, V1, V2∈R r×n V1 is an independent latent representation space for multi-object images and texts. T V2 is used for heterogeneous information interaction. Used to guide heterogeneous information interaction, Q1, Q2∈R r×r are projection matrices with orthogonal constraints for multi-object images and text data, B∈{-1,1} r×n is a joint hash code matrix whose elements are either -1 or 1, is the square of the F-norm of the matrix, I r is the identity matrix of dimension r, λ1, λ2, α, β1, β2, μ are trade-off parameters; The joint hash code matrix learning formula is solved by using the alternating update algorithm 1 to obtain the joint hash code matrix B; The alternating update algorithm 1 is summarized as follows: Construct a hash function learning formula for multi-object data graph-text matching problem; The hash function learning formula is: in, is the normalized category labeling matrix, F1, F2∈R r×q are the hash functions for multi-object images and text data, respectively, where r is the hash code length, q is the dimension of the kernelized data mentioned in the joint hash code matrix learning formula, and μ1 and μ2 are trade-off parameters; Use the alternating update algorithm 2 to solve the hash function learning formula and obtain hash functions F1 and F2; The alternating update algorithm 2 is summarized as follows: The joint hash code matrix B and hash functions F1 and F2 obtained in the above steps are used to test the test set and calculate the average accuracy of image-text matching in multi-object scenarios.

Citation Information

Patent Citations

  • Cross-modal data discrete hash retrieval method based on similarity maintenance

    CN110059198A

  • Image-text cross-modal retrieval method based on multi-class attention mechanism

    CN115658934A

  • Self-adaptive semi-pairing inquiry hash method for multi-modal retrieval

    CN117171393A

  • Image-text matching method for single-category low-information-amount data

    CN117315687A

  • Image-text processing method and device

    CN117893861A